Image processing device and method, and recorded medium

ABSTRACT

A storage picture generating unit ( 191 ) weight-adds an input picture and a storage picture to reduce the noise in a still picture portion. An area extraction unit ( 193 ) extracts a class tap from the input picture, while extracting a block from each of the input picture and the storage picture. A feature value detection unit ( 194 ) detects the dynamic range and the results of the waveform analysis from the class tap, while detecting the still/moving information of the subject pixel from the block. A class code detection unit ( 195 ) generates a class code which is based on the feature value. A table ( 196 ) outputs to an estimation calculating unit ( 198 ) sets of the prediction coefficients held by the table and which are associated with the class code. An area extraction unit ( 197 ) extracts prediction taps from the input picture and the storage picture. Using the sets of the prediction coefficients and the pixel information from the area extraction unit ( 197 ), the estimation calculating unit ( 198 ) sequentially generates pixel values y of an output picture.

TECHNICAL FIELD

This invention relates to a picture processing method and a pictureprocessing apparatus, and to a recording medium. More particularly, itrelates to a picture processing method and a picture processingapparatus, and to a recording medium, for satisfactorily generating apicture higher in quality than an input picture.

BACKGROUND ART

Up to now, a technique of performing variable processing on an inputpicture to generate a picture of higher picture quality has beendeveloped. By utilizing this technique, it is possible to generate, froma noisy input picture, a picture reduced in noise, or to generate apicture having its edge not blurred, from an input picture containing apicture area having relative movement with respect to other pictureportions, such as background, referred to below as a feature area, e.g.,a telop, and having the edge blurred.

Referring to FIGS. 1 to 9, the conventional method of generating apicture reduced in noise is explained.

In a conventional picture processing apparatus 10, a noisy input pictureis input to a substracter 11 and to an amplifier 14. The substracter 11is also fed from a frame memory 17 with a picture delayed one frame withrespect to the input picture. The picture from the frame memory 17 isalso supplied to an amplifier 16.

The substracter 11 subtracts the picture, sent from the frame memory 17,from the input picture, to send the resulting difference to a differencecalculation unit 12. The difference calculation unit 12 acquires theabsolute value of the results of the calculations from the substracter11, as a difference between the input picture and the picture from theframe memory 17 (picture of the directly previous frame picture) tooutput the so acquired absolute value to a threshold value decision unit13.

The threshold value decision unit 13 compares the difference value fromthe difference calculation unit 12 to a preset threshold value and,based on the results of comparison, decides whether or not the inputpicture is a still picture portion or a moving picture portion, frompixel to pixel, to set a weighting value p based on the result of thedecision.

If the input picture is determined to be the still picture portion, theweighting value p is set to a preset fixed value between 0 and 0.5. Ifthe input picture is determined to be the moving picture portion, theweighting value p is set to a value 1.

The amplifier 14 amplifies the signal of the input picture, with theweighting value p, as set by the threshold value decision unit 13, as anamplification factor, to send the amplified signal to an adder 15. Theamplifier 16 amplifies the signals of the picture from the frame memory17, using a value corresponding to subtraction of the weighting value pfrom unity (1), as an amplification factor, to send the resultingamplified signal to the adder 15, which then sums the outputs of theamplifiers 14, 16 together to output the resulting sum signal.

That is, if the input picture is the still picture portion, the pixelvalue of the input picture and the pixel value of a picture delayed byone frame from the input picture are weight-summed together, with aweighting value p. By summing the noisy input picture to the picture ofthe directly previous frame, in this manner, temporally non-steadyelements, such as noise, may be reduced.

However, if the input picture is a still picture portion, the weightingvalue p is set at a certain fixed value, and hence the noise cannot bereduced in association with its magnitude or direction. Moreover, if,due to mistaken decision by the threshold value decision unit 13, themoving picture portion is determined to be the still picture portion,weight-addition with the weighting value p is also applied to the movingpicture portion, so that, in this case, trailing type picture qualitydeterioration is produced.

If the input picture is the moving picture portion, the weight-additionwith the weighting value p=1 is executed. That is, the input picture isoutput directly, such that the noise is not reduced for the movingpicture portion. On the other hand, if the input picture is the stillpicture portion, but the noise contained therein is significant, thestill picture portion may be erroneously determined to be the movingpicture portion. In such case, the input picture (still picture portion)is directly output, that is, the noise is not reduced.

FIG. 2 shows another illustrative structure of a conventional pictureprocessing apparatus. In a picture processing apparatus 20, shown inFIG. 2, the noise of the moving picture portion is reduced byclassification adaptive processing which is based on taps correspondingto the motion vector.

The input picture, corrupted with noise, is sent to a frame memory 21-1.The frame memory 21-1 delays the input picture by one frame and sendsthe so delayed input picture to a frame memory 21-2, a motion vectordetection unit 22 and to area extraction units 24, 27.

The frame memory 21-2 delays the picture from the frame-memory 21-2 byone frame and sends the so delayed picture to a frame memory 21-3, amotion vector detection unit 23 and to the area extraction units 24, 27.

The frame memory 21-3 delays the picture from the frame-memory 21-2 byone frame and sends the so delayed picture to the motion vectordetection unit 23 and to the area extraction units 24, 27.

The motion vector detection unit 22 applies motion matching to twotemporally consecutive frames, sent from the frame memories 21-1, 21-2,to detect the motion vector to send the detected results to the areaextraction unit 24.

The motion vector detection unit 23 applies motion matching to twotemporally consecutive frames, sent from the frame memories 21-2, 21-3,to detect the motion vector to send the detected results to the areaextraction unit 27.

The area extraction unit 24 references the motion vector supplied fromthe motion vector detection unit 22 to extract preset picture areas asclass taps from the frames supplied from the frame memories 21-1 to21-3.

FIG. 3A shows picture areas extracted as class taps. A sum-total ofthree pixels, namely a subject pixel on a frame Fn from the frame memory21-2, as indicated with a black circle in the drawing, a pixel on aframe Fn−1 from the frame memory 21-1 (frame directly previous to theframe Fn), lying at a position specified by the motion vector from themotion vector detection unit 22 with respect to the subject pixel, asindicated by a hatched circle, and a pixel on a frame Fn+1 from theframe memory 21-3 (frame directly following the frame Fn), lying at aposition specified by the motion vector from the motion vector detectionunit 22 with respect to the subject pixel, as indicated by a hatchedcircle, are extracted as being class taps.

If, for example, the frame Fn and the other frames are as indicated inFIG. 3B, that is if the motion vector between the frame Fn−1 and theframe Fn is (−1, −1) and the motion vector between the frame Fn+1 andthe frame Fn is (1, 1), with the motion vector between the frame Fn andthe frame Fn being naturally (0, 0), as shown in FIG. 3B, the subjectpixel on the frame Fn, the pixel on the frame Fn−1 at a positionspecified by the motion vector (−1, −1) with respect to the subjectpixel and the pixel on the frame Fn+1 at a position specified by themotion vector (1, 1) with respect to the subject pixel, are extracted asbeing class taps, as shown in FIG. 3C.

The area extraction unit 24 sends the extracted class taps to a classcode generating unit 25.

The class code generating unit 25 applies e.g., ADRC processing to theclass taps supplied from the area extraction unit 24 and extractstempo-spatial patterns of the class taps while generating a class codeindicating the class obtained on classification conforming to theextracted patterns. The class code generating unit 25 sends thegenerated class code to a ROM table 26.

The ROM table 26 holds a set of prediction coefficients calculated for acase where a noisy picture is a pupil picture and a noise-free pictureis a teacher picture, from class to class, and outputs what correspondsto the class code sent from the class code generating unit 25, fromamong the prediction coefficients of the set, to an estimationprocessing unit 28.

The area extraction unit 27 references the motion vector supplied fromthe motion vector detection unit 23 to extract preset picture areas asprediction taps from picture data of the frames supplied from the framememories 21-1 to 21-3.

FIG. 4A shows the structure of the prediction taps. A subject pixel onthe frame Fn, indicated with a black circle in the drawing, along withpixels lying around the subject pixel, indicated with thin black in thedrawing, totaling at 13 pixels, a pixel on the frame Fn−1 specified bythe motion vector from the motion vector detection unit 22 with respectto the subject pixel, along with pixels lying around the pixel,indicated with thin black in the drawing, totaling at 13 pixels, and apixel on the frame Fn+1 specified by the motion vector from the motionvector detection unit 22 with respect to the subject pixel, along withpixels lying around the pixel, indicated with thin black in the drawing,totaling at 13 pixels, are extracted as being prediction taps.

If, for example, the motion vector between the frame Fn−1 and the frameFn is (−1, −1) and the motion vector between the frame Fn and the frameFn+1 is (1, 1), with the motion vector between the frame Fn and theframe Fn being naturally (0, 0), as shown in FIG. 4B, the subject pixelon the frame Fn, a pixel on the frame Fn−1 at a position specified bythe motion vector (−1, −1) with respect to the subject pixel and a pixelon the frame Fn+1 at a position specified by the motion vector (1, 1)with respect to the subject pixel, along with each 13 pixels therearound, are extracted as being class taps, as shown in FIG. 4C.

The area extraction unit 27 sends the extracted prediction taps to theestimation processing unit 28.

The estimation processing unit 28 executes preset calculations, based onthe prediction taps supplied from the area extraction unit 27 and on theset of the prediction coefficients, supplied from the ROM memory 26, togenerate a picture reduced in noise.

However, in the present instance, since the block matching is firstcarried out to detect the motion vector, a large amount of calculationsare needed, thus entailing extremely time-consuming operations.

FIG. 5 shows another illustrative structure of the conventional pictureprocessing apparatus. In a picture processing apparatus 30, shown inFIG. 5, classification adaptive processing based on taps correspondingto the motion vector is similarly carried out to reduce the noise in themoving picture portion. However, in the present case, the motion vectoris detected by a method in which the processing load is lesser than withthe block matching.

This picture processing apparatus 30 is provided with a tap arraydecision unit 31, which is to take the place of the motion vectordetections units 22, 23 of the picture processing apparatus 20 of FIG.2. It is noted that parts or components equivalent to those shown inFIG. 2 are denoted by the same base numerals to omit the correspondingdescription for simplicity.

A noisy input picture is sent to a frame memory 21-1. This frame memory21-1 delays the input picture by one frame and routes the so delayedinput picture to a frame memory 21-2, an area extraction unit 24, anarea extraction unit 27 and to a tap array decision unit 31.

The frame memory 21-2 delays the picture from the frame memory 21-1 byone frame and sends the so delayed picture to a frame memory 21-3, areaextraction units 24, 27 and to the tap array decision unit 31.

The frame memory 21-3 delays the picture from the frame memory 21-2 byone frame and sends the so delayed picture to a frame memory 21-4, areaextraction units 24, 27 and to the tap array decision unit 31.

The frame memory 21-4 delays the picture from the frame memory 21-3 byone frame and sends the so delayed picture to a frame memory 21-5, areaextraction units 24, 27 and to the tap array decision unit 31.

The frame memory 21-5 delays the picture from the frame memory 21-4 byone frame and routes the so delayed picture to the area extraction units24, 27 and to the tap array decision unit 31.

The tap array decision unit 31 detects the motion vector from the framememories 21-1 to 21-5, and decides the arraying positions of the classtaps or prediction taps, based on the so detected motion vector, to sendthe results of decision to the area extraction units 24, 27.

The motion vector is detected in the tap array decision unit 31 based onthe following premises:

In five consecutive frames, the interval between the first frame (framenumber 1) and the last frame (frame number five) is short, with thepicture movement over the five frames being a rectilinear movement at anequal speed.

If the noise is contained in none of the five consecutive frames, thevariance of pixels of the respective frames lying at the same positionof the picture is equal to 0 or extremely close to 0.

That is, in these premises, a straight line segment may be drawn whichpasses through a subject pixel on the frame Fn (frame from the framememory 21-3), and pixels lying in register with the subject pixel, thatis lying at the same positions on the picture, on the frame Fn−1 (framefrom the frame memory 21-2), frame Fn−2 (frame from the frame memory21-1), frame Fn+1 (frame from the frame memory 21-4) and frame Fn+2(frame from the frame memory 21-5), in a three-dimensional planecomprised of X axis and the Y axis on the picture and time, as shown inFIG. 6. That is, the tap array decision unit 31 detects this straightline segment as being a motion vector.

Referring to the flowchart of FIG. 7, the operation of the tap arraydecision unit 31, in case of executing the processing of tap arraydecision, is hereinafter explained.

At step S1, the tap array decision unit 31 sets, on the frame Fn, anarea comprised of, for example, 5×5 pixels, centered about a subjectpixel on the frame Fn from the frame memory 21-3.

At the next step S2, the tap array decision unit 31 selects a pixel fromthe area of the 5×5 pixels on the frame Fn, as set at step S1, as thecenter pixel, while setting a search range centered about a pixel on theframe Fn−1 from the frame memory 21-2, registering with the centerpixel.

At step S3, the tap array decision unit 31 initializes the coordinates(a, b), specifying a pixel in the search range, to (0, 0), as shown inFIG. 8.

At the next step S4, the tap array decision unit 31 generates a straightline segment passing through the pixel in the search range, specified bythe coordinates (a, b), that is a pixel on the frame Fn−1, and throughthe subject pixel on the frame Fn.

At step S5, the tap array decision unit 31 calculates the variancebetween the pixels of the frame Fn−2 to Fn+2, lying on the line segmentgenerated at step S4. At step S6, the tap array decision unit 31verifies whether or not the so calculated variance is smaller than thevariance values held at step S7, as later explained, that is whether ornot the variance calculated at step S6 is the smallest of the variancevalues calculated for straight line segments passing through the pixelsof the coordinates (a, b) checked so far and through the subject pixel.If the variance calculated is found to be of the least value, theprogram moves to step S7. At this step S7, the tap array decision unit31 holds the variance value calculated at step S5 and the associatedcoordinate (a, b).

If the variance calculated is found at step S6 not to be of the leastvalue, or if the variance value and the coordinates (a, b) are held atstep S7, the program moves to step S8, where the tap array decision unit31 updates the coordinates (a, b). The updating of the coordinates (a,b) may be done on a raster scan.

At the next step S9, the tap array decision unit 31 verifies whether ornot the coordinates (a, b) updated at step S8 exceed the search range asset at step S2 (FIG. 8). If it is verified that the search range is notexceeded, the program reverts to step S4 to execute the subsequentprocessing. If conversely the search range is found to be exceeded, thetap array decision unit 31 assumes that the search for the search rangehas been finished and proceeds to step S10 to vote for the coordinates(a, b) held at step S7.

At the next step S11, the tap array decision unit 31 checks whether ornot all of the pixels in the area as set at step S1 have been selectedout as being the center pixel If it is verified that not all of thepixels have been selected out as being the center pixel, the programreverts to step S2 to take out another pixel as being the center pixelto prosecute the following processing. If conversely all of the pixelshave been taken out as the center pixel, the program moves to step S12.

At step S12, the tap array decision unit 31 detects the coordinates (a,b) voted for the largest number of times at step S10. That is, thestraight line passing through the pixel on the frame Fn−1 specified bythe coordinates (a, b) and the subject pixel on the frame Fn is detectedto detect the straight line segment as the motion vector. The tap arraydecision unit 31 sends the position information of pixels of the framesFn−2 to Fn+2, lying on the so detected line segment, to the areaextraction units 24, 27.

This terminates the processing.

Since the preset straight line passing through the five consecutiveframes is detected as a motion vector, it is unnecessary to carry outthe voluminous calculations as needed in the case of block matching.

FIG. 9 shows another illustrative structure of the conventional pictureprocessing apparatus. In the picture processing apparatus 50, shown inFIG. 9, the noisy input picture is sent to each of a filter for reducingthe noise of the noisy still picture portion 51, a filter for reducingthe noise of the noisy moving picture portion 52 and to a motiondetection unit 53.

The filter for reducing the noise of the noisy still picture portion 51is made up of the components from the amplifier 14 to the frame memory17 of the picture processing apparatus 10 of FIG. 1, and performs weightaddition on the input picture and a picture delayed one frame from theinput picture. Thus, if the input picture is the still picture portion,the noise contained in the still picture portion may be reduced.

The filter for reducing the noise of the noisy moving picture portion 52is made up by the picture processing apparatus 20 shown in FIG. 2 or bythe picture processing apparatus 30 shown in FIG. 5, and executesclassification adaptive processing based on taps corresponding to themotion vector. So, if the input picture is the moving picture portion,the noise contained in the still picture portion may be reduced.

The motion detection unit 53 detects the motion from the input picture,on the pixel basis, and outputs the results of detection to an outputswitching unit 54.

If the result of detection from the motion detection unit 53 indicatesthat the input picture is the still picture portion, the outputswitching unit 54 selects an output of the filter for reducing the noiseof the noisy still picture portion 51 to route the so selected output tooutside. If conversely the result of detection from the motion detectionunit 53 indicates that the input picture is the moving picture portion,the output switching unit 54 selects an output of the filter forreducing the noise of the noisy moving picture portion 52 to route theso selected output to outside.

This reduces the noise contained in both the still picture portion andthe moving picture portion.

However, if, in the case of the present picture processing apparatus 50,the input picture is the still picture portion or the moving pictureportion, the processing by the filter for reducing the noise of thenoisy moving picture portion 52 or that by the filter for reducing thenoise of the noisy still picture portion 51 becomes redundant,respectively.

Consequently, the conventional picture processing apparatus 50 has adrawback that, as described above, the noise contained in the inputpicture cannot be reduced effectively.

Although it has been practiced to generate picture signals of thefourfold density from the input picture by e.g., linear interpolation,there is raised a problem that linear interpolation leads to a non-acutewaveform of the picture signals to give a blurred picture as the resultof the linear processing.

Meanwhile, a picture the edge of the feature area of which is notblurred may be generated by applying, for example, the classificationadaptive processing.

Referring to FIGS. 10 to 12, the picture processing of generating apicture, the edge of the feature area of which is not blurred, isexplained.

FIG. 10 shows an illustrative structure of a picture processingapparatus adapted for generating a picture the edge of the feature areaof which is not blurred. In a picture processing apparatus 60, shown inFIG. 10, the input picture, the edge of the feature area (picture arearelatively moving to the remaining picture area) of which is blurred, issent to the a feature area detecting portion 61 and to a classificationadaptive processing unit 62.

The feature area detecting portion 61 detects the feature area containedin the input picture to send the detected feature area to a synthesizingunit 63. Meanwhile, the feature area detecting portion 61 detects thefeature area so that the pixel density of the feature area will be equalto or higher than that of the input picture.

The classification adaptive processing unit 62 applies classificationadaptive processing for removing the noise, correcting the luminance orgenerating a high definition picture, to the input picture, and outputsthe resulting picture to the synthesizing unit 63.

The synthesizing unit 63 synthesizes the feature area from the featurearea detecting portion 61 to a picture from the classification adaptiveprocessing unit 62. By taking out the feature area once and synthesizingthe so taken out feature area to the remaining picture portion, it ispossible to generate a picture the edge of the feature area of which isnot blurred.

FIG. 11 shows an illustrative structure of the feature area detectingportion 61. The input picture is sent to a delay circuit 71 and to amotion vector detection circuit 75. The delay circuit 71 delays theinput picture, supplied thereto, by a time needed for processing bycircuits from the synthesizing circuit 72 to the phase shifting circuit76, to route the delayed input picture to the synthesizing circuit 72.This allows the synthesizing circuit 72, as later explained, tosynthesize the input picture to the corresponding picture.

The synthesizing circuit 72 synthesizes the input picture, suppliedthereto from the delay circuit 71, to the phase-shifted picture storedin a storage memory 73 and which is supplied from the phase shiftingcircuit 76. The synthesizing circuit 72 also routes the so synthesizedpicture to the storage memory 73.

The storage memory 73 stores the picture supplied from the synthesizingcircuit 72 to generate a storage picture, while routing the picture tothe detection circuit 74 and to the phase shifting circuit 76.

FIG. 12A shows typical level distribution of pixel values forming thefeature area on the stored picture. Thus, in the present instance, thelevel distribution of the feature area on the stored picture is the sameas that of the input picture, however, the level distribution of pixelvalues of the pixels making up the picture portion other than thefeature area is flattened out, as shown for example in FIG. 12B.

The detection circuit 74 detects the feature area from the storedpicture supplied from the storage memory 73 to route the so detectedfeature area to the synthesizing unit 63. Since the feature area on thestored picture has characteristics as explained with base to FIG. 12,the detection circuit 74 is able to detect the feature area accurately.

The motion vector detection circuit 75 is fed with the input picture andwith picture data and display positions of the feature area from thedetection circuit 74. The motion vector detection circuit 75 detects themotion vector between the feature area from the detection circuit 74 andthe feature area in the input picture to route the results of detectionto the phase shifting circuit 76.

The phase shifting circuit 76 phase-shifts the storage picture from thestorage memory 73, based on the motion vector from the motion vectordetection circuit 75, to send the phase-shifted picture to thesynthesizing circuit 72.

This allows to generate the picture the edge of the feature area ofwhich is not blurred. However, in synthesizing the feature area with theother picture portions, special processing needs to be carried out at aboundary portion, thus complicating the processing by the synthesizingunit 63.

The picture processing apparatus 60, configured as described above,suffers the problem that a picture of high picture quality, for example,a picture reduced in noise or a picture the edge of the feature area ofwhich is not blurred, cannot be produced satisfactorily from the inputpicture.

DISCLOSURE OF THE INVENTION

In view of the above depicted status of the art, it is an object of thepresent invention to enable a picture of high picture quality to beproduced satisfactorily from an input picture.

The present invention provides a picture processing apparatus forgenerating a second picture from a first picture, the second picturebeing of higher picture quality than the first picture, in which theapparatus includes acquisition means for acquiring the first picture,storage means for storing the first picture, acquired by the acquisitionmeans, storage processing means for storing a new first picture acquiredby the acquisition means at a position registering with the firstpicture stored in the storage means to permit a storage picture of thefirst picture to be stored in the storage means, first extraction meansfor extracting the first pixel information from both the storage pictureand the first picture, acquired by the acquisition means, based on theposition of a subject pixel of the second picture, feature valuedetection means for detecting a preset feature value from the firstpixel information, classification means for classifying the subjectpixel to one of a plurality of classes based on the feature value,second extraction means for extracting the second pixel information fromboth the storage picture and the first picture, acquired by theacquisition means, based on the position of the subject pixel, andgenerating means for generating the subject pixel by using the secondpixel information in accordance with a generating system presetcorresponding to the classes classified by the classification means.

The present invention also provides a picture processing method forgenerating a second picture from a first picture, the second picturebeing of higher picture quality than the first picture, in which themethod includes an acquisition step of acquiring the first picture, astorage step of storing the first picture, acquired by the acquisitionstep, a storage processing step of storing a new first picture acquiredby the acquisition step at a position registering with the first picturestored at the storage step to permit a storage picture of the firstpicture to be stored at the storage step, a first extraction step ofextracting the first pixel information from both the storage picture andthe first picture, acquired by the acquisition step, based on theposition of a subject pixel of the second picture, a feature valuedetection step of detecting a preset feature value from the first pixelinformation, a classification step of classifying the subject pixel toone of a plurality of classes based on the feature value, a secondextraction step of extracting the second pixel information from both thestorage picture and the first picture, acquired by the acquisition step,based on the position of the subject pixel, and a generating step ofgenerating the subject pixel by the second pixel information inaccordance with a generating system preset corresponding to the classesclassified by the classification step.

The present invention also provides a recording medium having recordedthereon a computer-readable program adapted for generating a secondpicture from a first picture, the second picture being of higher picturequality than the first picture, in which the program includes anacquisition step of acquiring the first picture, a storage step ofstoring the first picture, acquired by the acquisition step, a storageprocessing step of storing a new first picture acquired by theacquisition step at a position registering with the first picture storedat the storage step to permit a storage picture of the first picture tobe stored at the storage step, a first extraction step of extracting thefirst pixel information from both the storage picture and the firstpicture, acquired by the acquisition step, based on the position of asubject pixel of the second picture, a feature value detection step ofdetecting a preset feature value from the first pixel information, aclassification step of classifying the subject pixel to one of aplurality of classes based on the feature value, a second extractionstep of extracting the second pixel information from both the storagepicture and the first picture, acquired by the acquisition step, basedon the position of the subject pixel, and a generating step ofgenerating the subject pixel by the second pixel information inaccordance with a generating system preset corresponding to the classesclassified at the classification step.

The present invention also provides a picture processing apparatus forlearning preset data used in generating a second picture from a firstpicture, the second picture being higher in picture quality than thefirst picture, in which the apparatus includes generating means forgenerating a pupil picture equivalent to the first picture, storagemeans for storing the pupil picture, storage processing means forcausing a new pupil picture, generated by the generating means, to bestored at a position registering with the pupil picture stored in thestorage means for causing a storage picture of the pupil picture to bestored in the storage means, first extraction means for extracting thefirst picture information from both the storage picture and the pupilpicture generated by the generating means, based on the position of thesubject pixel of teacher data equivalent to the second picture, featurevalue detection means for detecting a preset feature value from thefirst pixel information, classification means for classifying thesubject pixel to one of a plurality of classes, second extraction meansfor extracting the second pixel information from both the storagepicture and the first picture generated by the generating means, basedon the position of the subject pixel and calculation means for findingthe preset data from one class of classifying by the classificationmeans to another, by using the second pixel information and the teacherdata.

The present invention also provides a picture processing method by apicture processing apparatus for learning preset data used in generatinga second picture from a first picture, the second picture being higherin picture quality than the first picture, in which the method includesa generating step of generating a pupil picture equivalent to the firstpicture, a storage step of storing the pupil picture, a storageprocessing step of causing a new pupil picture, generated by processingat the generating step, to be stored at a position registering with thepupil picture stored in the storage step for causing a storage pictureof the pupil picture to be stored in the storage step, a firstextraction step of extracting the first picture information from boththe storage picture and the pupil picture generated by the generatingstep, based on the position of the subject pixel of teacher dataequivalent to the second picture, a feature value detection step ofdetecting a preset feature value from the first pixel information, aclassification step of classifying the subject pixel to one of aplurality of classes, by way of classification, based on the featurevalue, a second extraction step of extracting the second pixelinformation from both the storage picture and the first picturegenerated by processing at the generating step, based on the position ofthe subject pixel, and a calculation step of finding the preset datafrom one class of classifying by the classification step to another, byusing the second pixel information and the teacher data.

The present invention also provides a recording medium having recordedthereon a computer-readable program for a picture processing apparatusadapted for learning preset data usable for generating a second picturefrom a first picture, the second picture being of higher picture qualitythan the first picture, in which the program includes a generating stepof generating a pupil picture equivalent to the first picture, a storagestep of storing the pupil picture, a storage processing step of causinga new pupil picture, generated by processing at the generating step, tobe stored at a position registering with the pupil picture stored in thestorage step of causing a storage picture of the pupil picture to bestored in the storage step, a first extraction step of extracting thefirst picture information from both the storage picture and the pupilpicture generated by the generating step, based on the position of asubject pixel of teacher data equivalent to the second picture, afeature value detection step of detecting a preset feature value fromthe first pixel information, a classification step of classifying thesubject pixel to one of a plurality of classes, a second extraction stepof extracting the second pixel information from both the storage pictureand the first picture generated by processing at the generating step,based on the position of the subject pixel, and a calculation step offinding the preset data from one class of classifying by theclassification step to another, by using the second pixel informationand the teacher data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an illustrative structure of aconventional picture processing apparatus.

FIG. 2 is a block diagram showing another illustrative structure of theconventional picture processing apparatus.

FIGS. 3A, 3B and 3C illustrate the structure of class taps in thepicture processing apparatus of FIG. 2.

FIGS. 4A, 4B and 4C illustrate the structure of prediction taps in thepicture processing apparatus of FIG. 2.

FIG. 5 is a block diagram showing another illustrative structure of theconventional picture processing apparatus.

FIG. 6 illustrates the method for detecting the motion vector in thepicture processing apparatus shown in FIG. 5.

FIG. 7 is a flowchart for illustrating the processing of tap arraydecision in the picture processing apparatus of FIG. 5.

FIG. 8 illustrates the search range.

FIG. 9 is a block diagram showing another illustrative structure of theconventional picture processing apparatus.

FIG. 10 is a block diagram showing an illustrative structure of apicture processing apparatus for generating a picture the edge of thefeature area of which is not blurred.

FIG. 11 is a block diagram showing an illustrative structure of afeature area detection unit in the picture processing apparatus of FIG.10.

FIGS. 12A and 12B show level distribution of pixels of the feature areaand those of the other picture portions.

FIG. 13 is a block diagram showing an illustrative structure of apicture processing apparatus embodying the present invention.

FIG. 14 is a block diagram showing an illustrative structure of astorage picture generating unit in the picture processing apparatus ofFIG. 13.

FIG. 15 is a block diagram showing another illustrative structure of astorage picture generating unit in the picture processing apparatus ofFIG. 13.

FIG. 16 is a block diagram showing still another illustrative structureof a storage picture generating unit in the picture processing apparatusof FIG. 13.

FIG. 17 is a block diagram showing an illustrative structure of an areaextraction unit in the picture processing apparatus of FIG. 13.

FIGS. 18A and 18B show a structure of a class tap in the pictureprocessing apparatus of FIG. 13.

FIG. 19 is a block diagram showing an illustrative structure of afeature value detection unit in the picture processing apparatus of FIG.13.

FIG. 20 is a block diagram showing another illustrative structure of anarea extraction unit in the picture processing apparatus of FIG. 13.

FIG. 21 is a block diagram showing another illustrative structure of apicture processing apparatus embodying the present invention.

FIG. 22 is a block diagram showing an illustrative structure of the areaextraction unit in the picture processing apparatus of FIG. 21.

FIG. 23 is a block diagram showing an illustrative structure of afeature value detection unit in the picture processing apparatus of FIG.21.

FIG. 24 shows the structure of class taps in the picture processingapparatus of FIG. 21.

FIGS. 25A, 25B show the structure of prediction taps in the pictureprocessing apparatus of FIG. 21.

FIGS. 26A, 26B show another level distribution of pixels of the featurearea and the other picture portions in case a high definition picture isto be generated from an input picture by synthesis as the pixels offeature areas of an input picture shifted to an out-of-phase positionrelative to those of a storage picture.

FIGS. 27A, 27B show another illustrative structure of class taps in thepicture processing apparatus of FIG. 21.

FIG. 28 shows another illustrative structure of prediction taps in thepicture processing apparatus of FIG. 21.

FIG. 29 is a flowchart showing the procedure of picture processingexecuted in the picture processing apparatus of FIG. 21.

FIG. 30 is a block diagram showing an illustrative structure of apicture processing apparatus for executing the learning processingembodying the present invention.

FIG. 31 is a block diagram showing an illustrative structure of a pupilpicture generating unit in the picture processing apparatus of FIG. 30.

FIG. 32 is a block diagram showing another illustrative structure of thepupil picture generating unit in the picture processing apparatus ofFIG. 30.

FIG. 33 is a block diagram showing still another illustrative structureof the pupil picture generating unit in the picture processing apparatusof FIG. 30.

FIGS. 34A, 34B illustrate a block extracted by the area extraction unitin the picture processing apparatus of FIG. 30.

FIG. 35 is a flowchart showing the procedure of the learning processingexecuted on the picture processing apparatus of FIG. 30.

FIG. 36 is a flowchart showing the procedure of generating a pupilpicture, executed in case the structure of FIG. 31 is used as the pupilpicture generating unit in the picture processing apparatus of FIG. 30.

FIG. 37 is a flowchart for illustrating the procedure of generating apupil picture, executed in case the structure of FIG. 32 is used as thepupil picture generating unit in the picture processing apparatus ofFIG. 30.

FIG. 38 is a flowchart for illustrating the procedure of generating apupil picture, executed in case the structure of FIG. 33 is used as thepupil picture generating unit in the picture processing apparatus ofFIG. 30.

FIG. 39 is a flowchart for illustrating the procedure of generating apupil picture, executed in case the teacher picture is higher inresolution than the pupil picture generated in the pupil picturegenerating unit in the picture processing apparatus of FIG. 30.

FIG. 40 is a flowchart showing the procedure of generating a storedpicture in case the structure of FIG. 14 is used as the pupil picturegenerating unit in the picture processing apparatus of FIG. 30.

FIG. 41 is a flowchart showing the procedure of generating a storedpicture in case the structure of FIG. 15 is used as the pupil picturegenerating unit in the picture processing apparatus of FIG. 30.

FIG. 42 is a flowchart showing the procedure of generating a storedpicture in case the structure of FIG. 16 is used as the pupil picturegenerating unit in the picture processing apparatus of FIG. 30.

FIG. 43 is a flowchart for illustrating the procedure of generating astored picture, executed in case the stored picture is higher inresolution than the pupil picture generated in the stored picturegenerating unit in the picture processing apparatus of FIG. 30.

FIGS. 44A, 44B illustrate a base picture and a four-fold density picturein the processing of motion vector detection in case of generating apicture (stored picture) higher in picture quality than the inputpicture (pupil picture).

FIG. 45 shows the relation between a base block and a reference block.

FIGS. 46A, 46B illustrate an absolute value sum table.

FIG. 47 is a block diagram showing the structure of a motion vectordetection device used for detecting the motion vector between the basepicture and a picture having four-fold density in the perpendiculardirection.

FIG. 48 is a block diagram showing an illustrative structure of acomputer operating as the aforementioned picture processing apparatus.

BEST MODE FOR CARRYING OUT THE INVENTION

Referring to the drawings, certain preferred embodiments of the presentinvention will be explained in detail.

FIG. 13 shows an illustrative structure of a picture processingapparatus 100 embodying the present invention. This picture processingapparatus 100 executes classification adaptive processing for noisereduction. In this classification adaptive processing, predictioncoefficients, calculated by the learning processing, as later explained,are used.

This picture processing apparatus 100 includes a storage picturegenerating unit 102, an area extraction unit 104 and an area extractionunit 107.

The storage picture generating unit 102 executes the processing ofeffectively reducing the noise contained in the still picture portion ofthe input picture or the processing of detecting the feature area.

FIGS. 14 to 19 show an illustrative structure of the storage picturegenerating unit 102.

FIG. 14 shows an exemplary structure of the storage picture generatingunit 102 in case of reducing the noise in the still picture portion. Anoisy pupil picture is sent to an amplifier 131, which amplifier 131amplifies signals of the pupil picture, supplied thereto, with a valueequal to unity (1) less a weighting value q (0<q<1), as an amplificationfactor, and routes the resulting amplified value to an adder 132.

From a storage picture memory 103, which will be explained subsequently,a storage picture generated immediately previously by the storagepicture generating unit 102, that is a storage picture previous to thenow supplied pupil picture by one frame, is sent to an amplifier 133,which amplifier 133 amplifies the signal of the storage picture, withthe weighting value q as the amplification factor, to route theresulting amplified picture to an adder 132.

The adder 132 sums an output of the amplifier 131 to an output of theamplifier 133 to route the resulting sum to the storage picture memory103.

That is, in this storage picture generating unit 102, the input pictureand the storage picture previous to the input picture by one frame areweight-added based on the weighting value q. For example, if the valueq=0.8, a storage picture is generated by adding 20% of the input pictureto 80% of the storage picture. By summing the input picture and thestorage picture preceding the input picture by one frame, at a presetproportion, the noise in the still picture portion can be reduced moreeffectively, as compared to the conventional picture processingapparatus 10 shown in FIG. 1.

Meanwhile, the storage picture, initially weight-added to the inputpicture, may be no other than the input picture initially stored in thestorage picture memory 103, or may be a picture obtained on averagingplural input pictures.

FIG. 15 shows another exemplary structure of the storage picture memory103 in case of reducing the noise in the still picture portion. Thisstorage picture generating unit 102 is made up of the storage picturegenerating unit 102 of FIG. 14 and a motion detection unit 141. Thismotion detection unit 141 is made up by components from a substracter142 to a threshold value decision unit 144, having the functions similarto those of the components from the substracter 11 to the thresholdvalue decision unit 13 forming the picture processing apparatus 10 shownin FIG. 1. That is, in this storage picture generating unit 102, theinput picture and the storage picture are weight-summed based on theweighting value q corresponding to the movement of the input picture.The weighting value q may also be set based on the difference inluminance between the input picture and the storage picture, in place ofon the picture movement.

FIG. 16 shows an illustrative structure of the storage picturegenerating unit 102 in case of detection of the feature area.

In this storage picture generating unit 102, the input picture is sentto a delay circuit 151 and to a motion vector detection circuit 155. Thedelay circuit 151 delays the input picture, supplied thereto, by timeneeded for processing in a synthesis circuit 152 and a phase shiftingcircuit 156, as later explained, to route the delayed input picture tothe synthesis circuit 152. This allows the synthesis circuit 152 tosynthesize the input picture and the associated picture together.

The synthesis circuit 152 sums the input picture, supplied from thedelay circuit 151, to the phase-shifted storage picture, stored in astorage memory 153 and which is supplied from the phase shifting circuit156. The synthesis circuit 152 sends the synthesized picture to thestorage memory 153.

The storage memory 153 stores the picture, supplied from the synthesiscircuit 152, to generate a stored picture, which is routed to andetection circuit 154 and to the phase shifting circuit 156.

The detection circuit 154 detects the feature area from the storagepicture, supplied from the storage memory 153, to send the so detectedfeature area to the motion vector detection circuit 155. Although thelevel distribution of the feature area on the storage picture is thesame as that of the input picture, the level distribution of pixelvalues of pixels making up a picture portion other than the feature areais flattened out, so that the detection circuit 154 is able to detectthe feature area high accurately.

The motion vector detection circuit 155 is fed with the input pictureand with the picture data and the display positions of the feature areafrom the detection circuit 154. The motion vector detection circuit 155detects the motion vector between the feature area from the detectioncircuit 154 and the feature area in the input picture to route theresults of detection to the phase shifting circuit 156.

Based on the motion vector from the motion vector detection circuit 155,the phase shifting circuit 156 phase-shifts the storage picture from thestorage memory 153 to route the resulting phase-shifted storage pictureto the synthesis circuit 152.

That is, with the present storage picture generating unit 102, thefeature area is detected, so that the input picture is stored in thestorage picture, as a preset phase relationship is maintained betweenthe feature area of the storage picture and the feature area of thestorage picture.

Meanwhile, the pixel density of the feature area, here generated, may behigher than or equivalent to that of the input picture. However, thecapacity of the storage picture memory 153 needs to be in meeting withthe pixel density of the feature area.

The above-described storage picture generating unit 102 routes thepicture reduced in noise and the feature area, thus generated anddetected, respectively, as the storage picture to the storage picturememory 103.

A storage picture generating unit 191 performs weight addition on theinput picture and the storage picture, based on the weighting value q,to reduce the noise in the still picture portion or to detect thefeature area. The picture or the feature area, thus reduced in the noisein the still picture portion, is sent as the storage picture to astorage picture memory 192.

The storage picture memory 192 stores the storage picture from thestorage picture generating unit 191, while suitably routing the storedstorage picture to area extraction units 193, 197.

The area extraction unit 104 extracts the class tap from the inputpicture, while extracting a block from each of the input picture and thestorage picture to route the so extracted class tap and blocks as thepixel information to a feature value detection unit 105.

FIG. 17 shows an illustrative structure of the area extraction unit 104.

In this area extraction unit 104, the input picture is routed to a framememory 16-1 and to a delay circuit 164, whilst the storage picture fromthe storage picture memory 103 is also routed to a block extraction unit166.

The frame memory 16-1 delays the pupil picture by one frame and routesthe so delayed pupil picture to a frame memory 16-2, a tap arraydecision unit 162 and to a class tap extraction unit 163.

The frame memory 16-2 delays the picture from the frame memory 16-1 byone frame and routes the so delayed picture to a frame memory 16-3, taparray decision unit 162 and to the class tap extraction unit 163.

The frame memory 16-3 delays the picture from the frame memory 161-2 byone frame and routes the so delayed picture to the frame memory 16-4,tap array decision unit 162 and to the class tap extraction unit 163.

The frame memory 16-4 delays the picture from the frame memory 161-3 byone frame and routes the so delayed picture to the frame memory 16-5,tap array decision unit 162 and to the class tap extraction unit 163.

The frame memory 16-5 delays the picture from the frame memory 16-4 byone frame and routes the so delayed picture to the tap array decisionunit 162 and to the class tap extraction unit 163.

Based on the picture data on the frame, supplied from the frame memories16-1 to 16-5, the tap array decision unit 162 decides the class taparraying positions, and routes the results of decision to the class tapextraction unit 163. Meanwhile, the processing of tap array decision,executed by the tap array decision unit 162, is similar to that executedby the tap array decision unit 31 shown in FIG. 5 and hence is not hereexplained in detail.

The storage picture from the storage picture memory 103 is sent to theclass tap extraction unit 163.

Referring to FIG. 18A, a subject pixel on the frame Fn, indicated with ablack circle in the drawing, and pixels on the frames Fn−2 to Fn+2,indicated by hatched circles, lying in position relationshipscorresponding to the position information from the tap array decisionunit 162 with respect to the subject pixel, are extracted as class taps.Also, the pixel on the storage picture lying at the same position as thesubject pixel on the frame Fn, also becomes the subject pixel, and isextracted as the class tap. FIG. 18B shows the storage picture suppliedfrom the storage picture memory 103 to the class tap extraction unit163.

This class tap extraction unit 163 outputs the class taps, extractedfrom the input picture, to the feature value detection unit 105.

The feature value detection unit 105 is configured as shown in FIG. 19.In this feature value detection unit 105, the class taps, extracted fromthe input picture by the area extraction unit 104, are sent to a DRanalysis unit 171 and to a waveform analysis unit 172. The class tap,extracted from the storage picture by the area extraction unit 104, issent to the waveform analysis unit 172,

The DR analysis unit 171 calculates the dynamic range of the pixel valueof the pixels, forming the class taps supplied, and converts its valueinto a binary number, which then is routed to a class code generatingunit 106.

The waveform analysis unit 172 performs waveform analysis simultaneouslyon the class taps extracted from the input picture and the class tapextracted from the storage picture.

For example, in a still picture, free of noise, no variations in thepixel values can occur from one frame to the next, as concerns the samepixel. In the case of a dynamic picture, deterioration in picturequality, such as blurring, may be noticed for a quick motion, however,there is basically no jitter, as concerns the same pixel. That is, ifvariations are noticed in the values of the same pixel, such variationsmay be regarded as being the noise. That is, waveform analysis of theclass taps from the pupil picture and the class tap from the storagepicture leads to detection of the noise contained therein.

Specifically, if the subject pixel is moving, there is produceddifference in luminance between the storage picture and the inputpicture. So, the results of the ADRC processing reveals the differencebetween the pixel values of the two pixels. For example, in the case ofthe one-bit ADRC, the values of the two pixels are (0, 1) or (1, 0). Onthe other hand, if the subject pixel is the still picture portion, thedifference in luminance is only negligible, so that the probability ishigh that, on ARC processing, the two pixel values become equal to eachother. In the case of one-bit ADRC, for example, the two pixel valuesare (0, 0) or (1, 1).

The feature value detection unit 105 uses this principle in such amanner that it executes still/moving discrimination by waveform analysisand detects the dynamic range and the results of waveform analysis fromthe class taps from the area extraction unit 104, while detecting theinformation as to whether the subject pixel is still or moving, from theblock of the area extraction unit 107, as the feature value needed forclassification. The feature value detection unit 105 sends the result ofdecision to the class code generating unit 106.

The class code generating unit 106 generates the class code, derivedfrom the feature value supplied from the feature value detection unit105, and routes the so generated class code to a ROM table 108.

The ROM table 108 holds a set of the prediction coefficients, calculatedfrom class to class by learning processing, which will be explainedsubsequently. The ROM table outputs, from the set of the predictioncoefficients, stored therein, those corresponding to the class code fromthe class code generating unit 106, to an estimation calculating unit109.

The area extraction unit 107 extracts prediction taps from the inputpicture and from the storage picture, to send the so extractedprediction taps to the estimation calculating unit 109.

FIG. 20 shows an exemplary structure of the area extraction unit 107.

In this area extraction unit 107, the input picture is sent to a framememory 18-1, while the storage picture from the storage picture memory103 is sent to a prediction tap extraction unit 183.

The frame memories 18-1 to 18-5 basically operate equivalently to theframe memories 16-1 to 16-5 of FIG. 17 and hence the correspondingdescription is here omitted for simplicity.

A tap array decision unit 182 decides the array positions of theprediction taps, based on the picture data of the frame sent from theframe memories 18-1 to 18-5, to send the results of decision to theprediction tap extraction unit 183. Meanwhile, the processing of taparray decision, to be performed by the tap array decision unit 182, issimilar to that performed by the tap array decision unit 31 of FIG. 5and hence is not here explained specifically.

With a pixel on the from supplied from the frame 18-3 as a subjectpixel, the prediction tap extraction unit 183 extracts pixels on theframe Fn−2 from the frame memory 18-1, frame Fn−1 from the frame memory18-2, frame Fn+1 from the frame memory 18-4 and the frame Fn+2 from theframe memory 18-5, lying in position relationships with respect to thesubject pixel, corresponding to the position information from the taparray decision unit 182, as prediction taps, to output the so extractedprediction taps to a normal equation adder 108. The prediction tapextraction unit 183 also renders the pixel on the storage pictureregistering with the subject pixel on the frame Fn a subject pixel, andextracts pixels on the storage picture, lying at preset positionrelationships with respect to the subject pixel, as prediction taps.

Using the set of the prediction coefficients from the ROM table 108 andpixel data derived from the pixel information from the area extractionunit 107, the estimation calculating unit 109 calculates the equation(1) to sequentially generate pixel values y of the output picture asbeing the results of the calculations.

That is, the estimation calculating unit 109 calculates the equation(1):y=w ₁ ×x ₁ +w ₂ ×x ₂ + . . . +w _(n) ×x _(n)  (1)which is a linear one-dimensional combination model prescribed by e.g.linear combination of pixel values of the extracted pixels x₁, . . . ,x_(n) and prediction coefficients w₁, . . . , w_(n), to find the pixelvalues of the output picture.

Meanwhile, poly-dimensional or non-linear equations, other than thelinear one-dimensional equation such as the equation (1), may also becalculated to find pixel values of the output picture.

FIG. 21 shows an exemplary structure of a picture processing apparatus190 embodying the present invention. This picture processing apparatus190 performs the processing of reducing the noise and correcting theedge of the feature area by classification adaptive processing. In thisclassification adaptive processing, prediction coefficients calculatedby undermentioned learning processing are used.

In this picture processing apparatus 190, a noisy input picture or aninput picture, the edge of the feature area of which is blurred, is sentto each of the storage picture generating unit 191 and area extractionunits 193, 197.

The storage picture generating unit 191 is configured similarly to thestorage picture generating unit 102 of the picture processing apparatus100 shown in FIG. 13, and performs weight addition, with the weightingvalue q, of the input picture and the storage picture, to reduce thenoise in the still picture portion or to detect the feature area. Thepicture having its still picture portion reduced in noise or thedetected feature area is sent as the storage picture to the storagepicture memory 192.

The storage picture memory 192 stores the storage picture from thestorage picture generating unit 191 and routes the stored storagepicture appropriately to the area extraction units 193, 197.

The area extraction unit 193 extracts class taps from the input picture,while extracting a block from each of the input picture and the storagepicture. The area extraction unit sends the so extracted class taps andblocks as pixel information to a feature value detection unit 194.

FIG. 22 shows an illustrative structure of the area extraction unit 193.

In this area extraction unit 193, the input picture is sent to the framememory 161-1 and to the delay circuit 164, while the storage picturefrom the storage picture memory 192 is sent to the block extraction unit166.

The frame memory 161-1 delays the pupil picture by one frame to routethe so delayed pupil picture to the frame memory 161-2, tap arraydecision unit 162 and to the class tap extraction unit 163.

The frame memory 161-2 delays the picture from the frame memory 161-1 byone frame to route the so delayed picture to the frame memory 161-3, taparray decision unit 162 and to the class tap extraction unit 163.

The frame memory 161-3 delays the picture from the frame memory 161-2 byone frame to route the so delayed picture to the frame memory 161-4, taparray decision unit 162 and to the class tap extraction unit 163.

The frame memory 161-4 delays the picture from the frame memory 161-3 byone frame to route the so delayed picture to the frame memory 161-5, taparray decision unit 162 and to the class tap extraction unit 163.

The frame memory 161-5 delays the picture from the frame memory 161-4 byone frame to route the so delayed picture to the tap array decision unit162 and to the class tap extraction unit 163.

A tap array decision unit 162 decides the array positions of the classtaps, based on the picture data of the frame sent from the framememories 161-1 to 161-5, to send the results of decision to theprediction tap extraction unit 163. Meanwhile, the processing of taparray decision, to be performed by the tap array decision unit 162, issimilar to that performed by the tap array decision unit 31 of FIG. 5and hence is not here explained specifically.

With a pixel on the frame Fn supplied from the frame 161-3 as a subjectpixel, the prediction tap extraction unit 163 extracts pixels on theframe Fn−2 from the frame memory 161-1 (frame preceding the frame Fn bytwo frames), frame Fn−1 from the frame memory 161-2 (frame preceding theframe Fn by one frame), frame Fn+1 from the frame memory 161-4 (framelater than the frame Fn by one frame) and the frame Fn+2 from the framememory 161-5 (frame later than the frame Fn by two frames), lying inposition relationships with respect to the subject pixel, correspondingto the position information from the tap array decision unit 162, asclass taps, to output the so extracted class taps to the feature valuedetection unit 105.

Since the arraying of the respective taps can be switched in this manneron the pixel basis, and the tap positions can be changed depending onthe motion vector, corresponding pixels can be extracted from frame toframe to allow to cope with movements.

The delay circuit 164 delays the pupil picture, sent thereto, in such amanner that a picture held by a frame memory 165 will be sent therefromto the block extraction unit 166 corresponding to the supply timing ofthe picture held in the frame memory 161-3 therefrom to the class tapextraction unit 163.

The block extraction unit 166 is fed not only with the picture from theframe memory 165 but also with the storage picture from the storagepicture memory 192.

The block extraction unit 166 extracts blocks of, for example, 8×8pixels, lying at the same positions of the input picture and in thestorage picture, to route the so extracted blocks to the feature valuedetection unit 194.

In this manner, the area extraction unit 193 extracts the class tapsfrom the input picture, while extracting blocks from the input pictureand the storage picture and routing the so extracted class taps andblocks as the pixel information to the feature value detection unit 194.

The feature value detection unit 194 detects preset feature value, fromthe pixel information supplied from the area extraction unit 193, suchas class taps or blocks, to route the so detected feature value to aclass code generating unit 195.

FIG. 23 shows an illustrative structure of the feature value detectionunit 194. In this feature value detection unit 194, the class tap,extracted by the class tap extraction unit 163 of the area extractionunit 193, is sent to the DR analysis unit 171 and to the waveformanalysis unit 172. The block extracted by the block extraction unit 166of the area extraction unit 197 is sent to a still/moving decision unit173.

The DR analysis unit 171 calculates the dynamic range of the pixelvalues of the pixels forming the class tap supplied thereto and convertsits value to a binary number which is routed to the class codegenerating unit 195.

The waveform analysis unit 172 performs 1-bit ADRC processing, forexample, based on pixel values of the pixels making up the class tapsupplied thereto, for waveform analysis. The waveform analysis unitsends the bit string, representing the results of analysis, to the classcode generating unit 195.

The still/moving decision unit 173 calculates the following equation(2): $\begin{matrix}{\text{differential~~value} = {\sum\limits_{X}{\sum\limits_{Y}{{{Y\left\lbrack {{in}\left( {x,y} \right)} \right\rbrack} - {Y\left\lbrack {{tmp}\left( {x,y} \right)} \right\rbrack}}}}}} & (2)\end{matrix}$from one supplied block to another to calculate the differential valueof the luminance of the pixels making up the block.

In the above equation, Y[in(x, y)] denotes a luminance value of a pixelon the block of the pupil picture specified by a coordinate (x, y) andY[tmp(x, y)] denotes a luminance value of a pixel of the block of thestorage picture specified by the coordinate (x, y).

On the other hand, the still/moving decision unit 173 checks whether ornot the calculated value of the difference is larger than a presetthreshold and, based on the result of decision, verifies whether thesubject pixel is still or moving. If the calculated value of thedifference is verified to be larger than the preset threshold value, thesubject pixel is determined to be moving so that the still/movingdecision unit 173 outputs the information specifying that effect, suchas the value 0, to the class code generating unit 195. If conversely thecalculated value of the difference is verified to be smaller than thepreset threshold value, the subject pixel is determined to be still, sothat the still/moving decision unit 173 outputs the informationspecifying that effect, such as the value 1, to the class codegenerating unit 195.

In this manner, the feature value detection unit 194 detects the dynamicrange and the results of the waveform analysis, from the class taps fromthe area extraction unit 193, while detecting the information specifyingwhether or not the subject pixel is still or moving, from the block ofthe area extraction unit 197, as the feature value necessary forclassification of the generated pixels, to send the resulting featurevalue to the class code generating unit 195.

In the present instance, the movement of the subject pixel representsthe feature value. Alternatively, the difference in luminance betweenthe block of the pupil picture and the block of the storage picture maybe used as the feature value. The difference in luminance between onepixels also suffices. Both the movement of the subject pixel and thedifference in luminance may be used as the feature values.

The class code generating unit 195 generates a class code, which isbased on the feature value supplied from the feature value detectionunit 194, to route the so generated class code to the ROM table 196.

The ROM table 196 holds the sets of the prediction coefficients, ascalculated by the above-mentioned learning processing, from class toclass, and outputs, from the sets of the prediction coefficients, storedtherein, those corresponding to the class code from the class codegenerating unit 195 to an estimation calculating unit 198.

The area extraction unit 197 is configured similarly to the areaextraction unit 107 of the picture processing apparatus 100, shown inFIG. 13, that is, is configured as shown in FIG. 20, and extractsprediction taps from the input picture and the storage picture to sendthe so extracted prediction taps to the estimation calculating unit 198.

Using the sets of the prediction coefficients from the ROM table 196,and the picture data based on the pixel information from the areaextraction unit 197, the estimation calculating unit 198 calculates theabove-mentioned equation (1), and sequentially generates the pixelvalues y of the output picture as being the results of the calculations.

The operation for generating a picture reduced in noise in theabove-described picture processing apparatus 190 is hereinafterexplained.

The noisy input picture is sent to the storage picture generating unit191, area extraction unit 193 and to the area extraction unit 197.

In the present case, the storage picture generating unit 191 isconfigured similarly to the storage picture generating unit 102 of thepicture processing apparatus 100, that is, configured similarly to thesetup shown in FIG. 14 or FIG. 15. That is, the storage picturegenerating unit 191 generates the storage picture as it effectivelyreduces the noise of the still picture portion. The storage picturegenerated is sent to and stored in the storage picture memory 192.

The area extraction unit 193 extracts the class tap from the inputpicture, while extracting blocks from the input picture and the storagepicture and routing them to the feature value detection unit 194.

In the present instance, a subject pixel on the frame Fn, indicated witha black circle, and the pixels on the frame Fn−2, frame Fn−1, frame Fn+1and the frame Fn+2, indicated with hashed circles, lying in positionrelationships with respect to the subject pixel corresponding to theposition information from the tap array decision unit, totaling at fivepixels, are extracted as class taps, as shown in FIG. 24.

The feature value detection unit 194 calculates the dynamic range of thepixel values of the pixels, making up the class tap, based on the classtaps from the area extraction unit 193, and performs waveform analysisby the 1-bit ADRC processing, while making still/moving decision for thesubject pixel based on the block from the area extraction unit 193.

The feature value detection unit 194 sends the dynamic range, results ofthe waveform analysis and the results of the still/moving decision tothe class code generating unit 195.

The class code generating unit 195 generates a class code, which isbased on data from the feature value detection unit 194, to route the sogenerated class code to the ROM table 196.

The ROM table 196 outputs, from among the sets of the predictioncoefficients for noise reduction, stored from class to class, thoseassociated with the class code from the class code generating unit 195,to the estimation calculating unit 198.

The area extraction unit 197 extracts the prediction taps from the inputpicture and the storage picture to route the so extracted predictiontaps to the estimation calculating unit 198.

In the present instance, a subject pixel on the frame Fn, indicated witha black circle, and each 13 pixels on the frames Fn−2 to Fn+2, indicatedby hashed circles, lying in position relationships with respect to thesubject pixel which are based on the position information from the taparray decision unit, are extracted as prediction taps, as shown in FIG.25A. In addition, the pixel on the storage picture registering with thesubject pixel on the frame Fn is rendered the subject pixel as indicatedwith a black circle in FIG. 25B, whilst the pixels on the storagepicture, indicated with hashed circles, lying in preset positionrelationships with respect to the subject pixel, are also extracted asprediction taps.

Using the sets of the prediction coefficients from the ROM table 196,and pixel data of the prediction taps which are based on the pixelinformation from the area extraction unit 197, the estimationcalculating unit 198 sequentially calculates the aforementioned equation(1) to sequentially generate pixel values y of the output picture.

Next, the operation of generating a picture, the edges of the featurearea of which are not blurred, by the present picture processingapparatus 190, is now explained.

In this instance, the edges of the feature area are corrected bygenerating a picture of high definition from the input picture. That is,in the storage picture generating unit 191, the input picture and thestorage picture are synthesized in such a manner that the pixels of thefeature areas thereof are out of phase relative to each other a presetamount, as shown in FIG. 26A, whereby the feature area as the storagepicture is detected. By so doing, the storage picture becomes a highdefinition picture having the same pixel density as that of anultimately produced picture, as shown in FIG. 26B.

The operation of the area extraction unit 193 is basically the same asthat in case of noise reduction, and hence is not explained specificallyfor simplicity. The class tap in the present instance is of a structuresuch as is shown in FIG. 27.

That is, the class taps are made up of eight pixels on the frame Fn,lying at a preset position relationship with a subject pixel, indicatedby a hashed circle, of the ultimately produced picture (pictureconcurrent with the frame Fn and with the storage picture), each fivepixels of the frames Fn−2, Fn−1, Fn+1 and Fn+2 (FIG. 27A) and 11 pixelsof the storage picture.

Meanwhile, if a pixel of a picture generated between the pixels of theinput picture is a subject pixel, the pixels on the frames Fn−2 to Fn+2,extracted as class taps, are not unchanged, as shown in FIG. 28, nomatter which or the pixels is the subject pixel. The pixels extractedfrom the storage picture differ with the positions of the subject pixel.

Although the class taps are different in structure from the predictiontaps, the the class taps and the prediction taps may also be the of thesame structure.

In the above-described picture processing apparatus 190, the pictureprocessing is carried out in accordance with the procedure shown in theflowchart shown in FIG. 29.

That is, in the present picture processing apparatus 190, an inputpicture is first acquired at step S111.

At the next step S112, a feature area, reduced in noise in the stillpicture portion, is detected from the input picture, by the storagepicture generating unit 191, so as to be stored as a storage picture inthe storage picture memory 192.

At the next step S113, the class tap is extracted from the input pictureby the area extraction unit 193, whilst a block is extracted from eachof the input picture and the storage picture.

At the next step S114, a preset feature value is detected from the inputpicture by the feature value detection unit 194, based on the class tapand the blocks extracted from the area extraction unit 193.

At the next step S115, the class code is generated by the class codegenerating unit 195, based on the feature value detected by the featurevalue detection unit 194.

At the next step S116, the sets of the prediction coefficients,corresponding to the class code generated by the class code generatingunit 195, are output from the ROM table 196.

At the next step S117, predictive calculations are carried out by theestimation calculating unit 198, using the sets of the predictioncoefficients from the ROM table 196, and the picture data of theprediction taps, which are based on the picture information from thearea extraction unit 197.

At the next step S118, pixel values of the output picture, obtained onpredictive calculations by the estimation calculating unit 198, aresequentially output.

In the present picture processing apparatus 190, it is checked at thenext step S118 whether or not the totality of the pixel values of theoutput picture have been output. If there is any pixel value not as yetoutput, the program reverts to step S111 to repeat the processing asfrom the step S111 to the step S118 to output the totality of the pixelvalues of the output picture to terminate the processing.

FIG. 30 shows an exemplary structure of a picture processing apparatus300 embodying the present invention. This picture processing apparatus300 executes the learning processing for finding the predictioncoefficients used in the processing of picture generation in theabove-described picture processing apparatus 100, 190.

In this picture processing apparatus 300, a picture which is to be ateacher picture in learning, for example, a picture free of noise or apicture, the edges of the feature area of which are not blurred, issupplied to a pupil picture generating unit 301 and to a normal equationaddition unit 308.

The pupil picture generating unit 301 processes the teacher picture in apreset fashion to generate a pupil picture corresponding to the inputpicture in the processing of picture generation. For example, noise issuperimposed on the teacher picture to generate a pupil picture usablefor calculating prediction coefficients in the processing of generatinga picture reduced in noise, or the teacher picture is deteriorated inresolution, such as by decimation, to generate a pupil picture usablefore calculating the prediction coefficients in the processing ofgenerating a picture the edge of the feature area of which is notblurred.

For example, if noise is superimposed on the teacher picture, the pupilpicture generating unit 301 generates the random noise to add it to theteacher picture. The noise may also be superimposed on the teacherpicture by a configuration shown for example in FIGS. 31 to 33.

In the case of the pupil picture generating unit 301, shown in FIG. 31,an RF modulation unit 321 generates random noise, which is routed to anattenuator 322. The attenuator 322 attenuates the output of the RFmodulation unit 321 to output the resulting attenuated output to an RFdemodulation unit 323. The RF demodulation unit 323 RF demodulates theoutput of the attenuator 322 to generate a picture corresponding to theteacher picture with the noise superimposed thereon.

In the case of the pupil picture generating unit 301, shown in FIG. 32,a picture with a uniform background is sent to the RF modulation unit321 and to a substracter 324. The processing by the components from theRF modulation unit 321 to the RF demodulation unit 323 is not explainedhere since it is the same as the processing explained with reference toFIG. 32. An output of the RF demodulation unit 323 is sent to thesubstracter 324. The substracter 324 calculates the difference between apicture with a uniform background and the output of the RF demodulationunit 323 to output the so calculated difference to an adder 325. Theadder 325 sums the output of the substracter 324 to the teacher data togenerate a picture corresponding to the teacher picture with the noisesuperimposed thereon.

In the pupil picture generating unit 301, shown in FIG. 33, the picturewith a uniform background is sent to the RF modulation unit 321. Theprocessing by the components from the RF modulation unit 321 to the RFdemodulation unit 323 is not explained in detail since it is the same asthat explained in connection with FIG. 31. An output of the RFdemodulation unit 323 is sent to the substracter 324 and to a frameaddition circuit 326. The frame addition circuit 326 generates a picturecorresponding to an output of the RF demodulation unit 323 less thenoise, by addition of a frame supplied from the RF demodulation unit323, to send the so generated picture to the substracter 324. Theprocessing by the substracter 324 and the adder 325 is not explainedhere since it is the same as the processing explained in connection withFIG. 32.

In the present picture processing apparatus 300, a storage picturegenerating unit 302, similar in structure to the storage picturegenerating unit 102 or 191 of the picture processing apparatus 100 shownin FIG. 13 or the picture processing apparatus 190 shown in FIG. 21,that is the configuration shown in FIGS. 14 to 16, is used. In thepresent storage picture generating unit 302, weight addition with theweighting value of q is applied to the pupil picture and to the storagepicture of the directly previous frame. For example, if the value q=0.8,a storage picture is generated by adding 20% of the pupil picture to 80%of the storage picture. By summing the pupil picture and the storagepicture preceding the pupil picture by one frame, at a presetproportion, the noise in the still picture portion can be reducedeffectively. Moreover, since the pupil picture and the storage pictureof the directly previous frame are stored and summed together at apreset proportion, the noise of the still picture portion can be reducedmore effectively.

The storage picture generating unit 302 sends the picture so generated(detected) and which is reduced in noise as a storage picture to astorage picture memory 303.

The storage picture memory 303 holds a picture from the storage picturegenerating unit 302 and appropriately routes the storage picture to thearea extraction units 304, 307.

The area extraction unit 304 extracts the pixel information, necessaryfor classification, from the pupil picture from the pupil picturegenerating unit 301 and the storage picture from the storage picturememory 303, to route the so extracted information to the feature valuedetection unit 305. The area extraction unit 304 may be of the structuresimilar to that of the area extraction unit 193 in the above-describedpicture processing apparatus 190, that is, may be of the structure shownin FIG. 22. This area extraction unit 304 extracts the class taps fromthe pupil picture, while extracting the blocks of, for example, 8×8pixels, lying at the same positions of the pupil picture and the storagepicture, as shown in FIGS. 34A and 34B, to send the so extracted classtaps and blocks as the pixel information to the feature value detectionunit 305.

The feature value detection unit 305 detects the dynamic range and theresults of the waveform analysis, from the class taps from the areaextraction unit 304, while detecting the information specifying whetheror not the subject pixel is still or moving, from the block of the areaextraction unit 304, as the feature value necessary for classificationof the generated pixels, to send the resulting feature value to theclass code generating unit 306. The feature value detection unit 305 maybe of the structure similar to that of the feature value detection unit105 used in the picture processing apparatus 100, that is of thestructure shown in FIG. 23.

In the present instance, the motion of the subject pixel is used as thefeature value. Alternatively, the difference in luminance between theblock of the pupil picture and that of the stored picture may be used asthe feature value. The difference of luminance between one pixels mayalso suffice. It is also possible to use both the motion of the subjectpixel and the different in luminance.

The class code generating unit 306 generates a class code, based on thefeature value from the feature value detection unit 305, to send the sogenerated class code to the normal equation addition unit 308.

The area extraction unit 307 extracts the pixel information, needed forprediction, from the pupil picture from the pupil picture generatingunit 301 and from the storage picture from the storage picture memory303, and routes the pixel information so extracted to the normalequation addition unit 308. As this area extraction unit 307, the sametype of the unit as the area extraction unit 107 in the above-describedpicture processing apparatus 100, shown in FIG. 20, is used.

Based on the class code from the class code generating unit 306, teacherpicture and on the prediction taps from the area extraction unit 307,the normal equation addition unit 308 calculates data needed for solvingthe normal equation to find the prediction coefficients, and outputs thecalculated results to a prediction coefficient decision unit 309.

Using the calculated results from the normal equation addition unit 308,the prediction coefficient decision unit 309 executes presetcalculations to find class-based prediction coefficients to send the socalculated prediction coefficients to a memory 310 for storage therein.

The calculations executed in the normal equation addition unit 308 andin the prediction coefficient decision unit 309 are now explained.

In the above-described picture generation processing, the subject pixeland the pixels lying at preset position relationships with respect tothe subject pixel are extracted from the input picture (noisy inputpicture or an input picture the edge of the feature area of which isblurred) and from the storage picture (the storage picture the noise inthe still picture portion of which is reduced or the storage picture asthe detected feature area). The above equation (1), which is the linearone-dimensional model defined by the linear combination of the values ofthe extracted pixels (pupil data) x₁, . . . , x_(n) and the predictioncoefficients w₁, . . . , w_(n), as calculated by the learningprocessing, is calculated to find pixel values of the output picture.

Meanwhile, poly-dimensional or non-linear equations, instead of thelinear one-dimensional equations, maybe calculated to find the pixelvalues of the output picture.

Although the equation (1) may be represented as the equation (3), theprediction coefficients w are not uniquely determined in case k=1, 2, 3,. . . , m in the equation (3), where m>n. Thus, in such case, theprediction coefficients w are found by the so-called least squaremethod:y _(k) =w ₁ ×x _(k1) +w ₂ ×x _(k2) + . . . +w _(n) ×x _(kn)  (3).

That is, the prediction coefficients are found so that the value of theequation (5) when the element e_(k) of the error vector e is defined bythe equation (4):e _(k) =y _(k)−(w ₁ ×x _(k1) +w ₂ ×x _(k2) + . . . +w _(n) ×x_(kn))  (4)e²=Σe_(k) ²  (5)will be of the smallest value.

By way of explanation of the least square method more specifically, e²is partially differentiated with respect to the prediction coefficientsw_(i) (i=1, 2, . . . ), as indicated by the equation (6):$\begin{matrix}{\frac{\partial e^{2}}{\partial w_{i}} = {{\sum\limits_{k = 0}^{m}{2\left( \frac{\partial e_{k}}{\partial w_{i}} \right)e_{k}}} = {\sum\limits_{p = 0}^{m}{2{x_{ki} \cdot {e_{k}.}}}}}} & (6)\end{matrix}$

If the pupil data x_(ji), made up of pixel values of the extracted inputpicture or storage picture, are defined as indicated in the equation(7): $\begin{matrix}{x_{ji} = {\sum\limits_{p = 0}^{m}{x_{pi} \cdot x_{pj}}}} & (7)\end{matrix}$and the pixel values of the teacher picture y_(i) (teacher data) aredefined as indicated in the equation (8): $\begin{matrix}{y_{i} = {\sum\limits_{k = 0}^{m}{x_{ki} \cdot y_{k}}}} & (8)\end{matrix}$the equation (6) may be expressed by a matrix usually termed a normalequation indicated by the equation (9): $\begin{matrix}{{\begin{pmatrix}x_{11} & x_{12} & \cdots & x_{1n} \\x_{21} & x_{22} & \cdots & x_{2n} \\\cdots & \cdots & \; & \cdots \\x_{n1} & x_{n2} & \cdots & x_{nn}\end{pmatrix}\begin{pmatrix}w_{1} \\w_{2} \\\cdots \\w_{n}\end{pmatrix}} = \begin{pmatrix}y_{1} \\y_{2} \\\cdots \\y_{n}\end{pmatrix}} & (9)\end{matrix}$it being noted that the pupil data x_(ji) means the number j row pupildata in the set of number i column pupil data (set of pupil data usedfor predicting the number i column pupil data y_(i)).

That is, the normal equation addition unit 308 calculates the equations(7) and (8), based on the pupil picture, storage picture and the teacherdata, supplied thereto, to calculate the pupil data x_(ij) formed by thevalues of pixels of the pupil picture or the storage picture and teacherdata y_(i) of pixel values of the teacher picture. Based on the routinematrix solution method, such as the sweep-out method, the predictioncoefficient decision unit 309 solves the normal equation (9) tocalculate the prediction coefficients w.

The above-described picture processing apparatus 300 executes pictureprocessing in accordance with the flowchart shown in FIG. 35.

That is, in the present picture processing apparatus 300, the teacherdata is first acquired at step S121.

At the next step S122, the pupil picture generating unit 301 performspreset processing on the teacher picture acquired at step S121 togenerate a pupil picture equivalent to the input picture in theprocessing of picture generation.

At the next step S123, the pupil picture and the storage picture of thedirectly previous frame are weight-summed in the storage picturegenerating unit 302 to detect from the pupil picture the feature areacorresponding to the still picture portion reduced in noise. The featurearea so detected is stored as a storage picture in the storage picturememory 303.

At the next step S124, the class taps are extracted as the pixelinformation required for classification by the area extraction unit 304from the pupil picture generated by the pupil picture generating unit301, at the same time as a block formed by plural pixels lying at theregistering positions of the pupil picture and the storage picture isalso extracted.

At the next step S125, the feature value detection unit 305 acquires thedynamic range and the results of the waveform analysis from the classtaps extracted from the area extraction unit 304, as feature valueneeded for classification of the generated pixels, while acquiring, fromthe blocks extracted by the area extraction unit 304, the informationindicating whether the subject pixel is still or moving.

At the next step S126, the class code is generated by the class codegenerating unit 306 based on the feature value detected by the featurevalue detection unit 305 At the next step S127, data needed to solve thenormal equation to find the prediction coefficients are calculated basedon the class code from the class code generating unit 306, teacherpicture and on the prediction taps from the area extraction unit 307.

At the next step S128, the prediction coefficient decision unit 309executes preset calculations, using the results of calculations from thenormal equation addition unit 308, to decide class-based predictioncoefficients.

At the next step S129, the prediction coefficients, calculated by theprediction coefficient decision unit 309 from class to class, are storedin the memory 310.

In this picture processing apparatus 300, it is checked at the next stepS130 whether or not the processing has been done on all of the picturedata. Should there be any data to be processed, the program reverts tostep S121 to repeat the processing as from step S121 to step S130 toprocess the totality of the picture data to terminate the learningprocessing.

If the configuration of FIG. 31 is used as the present pupil picturegenerating unit 301, the processing of generating the pupil picture atthe above step S122 is carried out in accordance with the procedure ofthe flowchart of FIG. 36.

That is, the present pupil picture generating unit 301 at step S131directly adds the noise to the teacher picture by the attenuator 322.

At the next step S132, the teacher picture, summed with the noise by theprocessing of step S131, is output as pupil picture.

At the next step S133, it is checked whether or not the totality of thepicture data for one field has been processed. If there is any data forprocessing, the program reverts to step S131 to repeat the processingfrom step S131 to step S133 to process the totality of the picture datato terminate the processing.

If the configuration of FIG. 32 is used as the present pupil picturegenerating unit 301, the processing of generating the pupil picture atthe above step S122 is carried out in accordance with the procedure ofthe flowchart of FIG. 37.

Specifically, the pupil picture generating unit 301 at step S141 addsthe noise to a picture having a uniform background by the attenuator322.

At the next step S142, the original picture with the uniform backgroundis subtracted by substracter 324 from the picture with the uniformbackground, which is summed with the noise by the processing at stepS141, thereby extracting only the noise component attributable to theattenuator 322.

At the next step S143, the noise extracted by the processing of theabove step S143 is added to the teacher picture by the adder 325.

At the next step S144, the teacher picture, summed with the noise by theprocessing of step S143, is output as pupil picture.

At the next step S145, it is checked whether or not the totality of thepicture data for one field has been processed. Should there be anypicture data left for processing, the program reverts to step S141 torepeat the processing as from step S141 to S145 to process the totalityof the picture data to terminate the processing.

If the configuration of FIG. 33 is used as the pupil picture generatingunit 301, the processing of generating the pupil picture at the abovestep S122 is carried out in accordance with the procedure of theflowchart shown in FIG. 38.

That is, the present pupil picture generating unit 301 at step S151 addsthe noise to a picture of the uniform background by the attenuator 322.

At the next step S152, frame addition by the frame addition circuit 326is applied to the picture of the uniform background, which is summedwith the noise by the processing of the above step S151, to generate anoise-free picture.

At the next step S153, the noise-free picture obtained by the processingat step S151 is subtracted by the substracter 324 from the picture withthe uniform background, which is summed with the noise by the processingof the above step S151, thereby extracting only the noise componentattributable to the attenuator 322.

At the next step S154, the noise extracted by the processing at stepS153 is added to the teacher picture by the adder 325.

At the next step S155, the teacher picture summed with the noise by theprocessing at the above step S153 is output as the pupil picture.

At the next step S156, it is checked whether or not the totality of thepicture data for one field has been processed. Should there be anypicture data left for processing, the program reverts to step S151 torepeat the processing as from step S151 to S156 to process the totalityof the picture data to terminate the processing.

If, in the processing of generating the pupil picture at the above stepS122, the teacher picture is higher in resolution than the pupil pictureto be generated, the pupil picture is generated in accordance with theprocedure of the flowchart of FIG. 39 to generate the pupil picture.

That is, the pupil picture generating unit 301 at step S161 applies theprocessing of deteriorating the resolution of the pupil picture bydecimation or filtering to the teacher data.

At the next step S162, the teacher picture, deteriorated in resolutionby the processing at step S161 is output as a pupil picture.

At step S163, it is verified whether or not the totality of the picturedata for one field has been processed. Should there be any picture dataleft for processing, the program reverts to step S161 to repeat theprocessing as from step S161 to S163 to process the totality of thepicture data to terminate the processing.

If the configuration of FIG. 14 is used as the pupil picture generatingunit 302, the processing of generating the storage picture at the abovestep S123 is carried out in accordance with the procedure of theflowchart shown in FIG. 40.

Specifically, the present pupil picture generating unit 302 at step S171acquires the storage picture of the directly previous field from thestorage picture memory 303.

At the next step S172, the storage picture of the directly previousfield is multiplied with a weighting value q, while the pupil picture ismultiplied with a weighting value (1−q).

At the next step S173, the storage picture of the directly previousframe and the pupil picture, multiplied at the step S172 with theweighting values (q, 1−q), are summed and synthesized together to form anew storage picture.

At the next step S174, the new storage picture, synthesized at the abovestep S173, is output.

At step S175, it is verified whether or not the totality of the picturedata for one field has been processed. Should there be any picture dataleft for processing, the program reverts to step S171 to repeat theprocessing as from step S171 to S175 to process the totality of thepicture data to terminate the processing.

If the configuration of FIG. 15 is used as the pupil picture generatingunit 302, the processing of generating the storage picture at the abovestep S123 is carried out in accordance with the procedure of theflowchart shown in FIG. 41.

Specifically, the present storage picture generating unit 302 at stepS181 acquires the storage picture of the directly previous field fromthe storage picture memory 303.

At the next step S182, the motion detection unit 141 takes thedifference between the registering pixels to make still/moving decision.

At the next step S183, the weighting value q is determined on the basisof the difference between the registering pixels, based on thedifference value of the registering pixels as calculated at step S182.

At the next step S184, the storage picture of the directly previousfield is multiplied with the weighting value q, while the pupil pictureis multiplied with the weighting value 1−q.

At the next step S185, the storage picture of the directly previousfield and the pupil data, multiplied at step S184 with the weightingvalues q and 1−q, respectively, are summed and synthesized together toform a new storage picture.

At the next step S186, a new storage picture, synthesized at the abovestep S185, is output.

At the next step S187, it is verified whether or not the totality of thepicture data for one field has been processed. Should there be anypicture data left for processing, the program reverts to step S181 torepeat the processing as from step S181 to S187 to process the totalityof the picture data to terminate the processing.

If the configuration of FIG. 16 is used as the pupil picture generatingunit 302, the processing of generating the storage picture at the abovestep S123 is carried out in accordance with the procedure of theflowchart shown in FIG. 42.

That is, the storage picture generating unit 302 at step S191 acquiresthe storage picture of the previous field from the storage picturememory 303.

At the next step S193, a feature area is detected by an detectioncircuit 154 from the storage picture supplied from the storage memory153.

At the next step S193, the equal density motion vector is detected basedon the feature area detected at step S192. That is, the motion vectordetection circuit 155 detects the motion vector between the feature areafrom the detection circuit 154 and the feature area in the pupilpicture.

At the next step S194, the storage picture from the storage memory 153is phase-shifted, based on the motion vector detected at step S193, bythe phase shifting circuit 156, to effect position matching with thepupil picture.

At the next step S195, the storage picture and the pupil picture,position-matched at step S194, are summed and synthesized together toform a new storage picture.

At the next step S196, the new storage picture, synthesized at the abovestep S195, is output.

At the next step S197, it is verified whether or not the totality of thepicture data for one field has been processed. Should there be anypicture data left for processing, the program reverts to step S191 torepeat the processing as from step S191 to S197 to process the totalityof the picture data to terminate the processing.

If, in the processing of generating the storage picture at step S123,the storage picture is higher in resolution than the pupil picture to begenerated, the storage picture is generated in accordance with theprocedure of the flowchart shown in FIG. 43.

Specifically, the present storage picture generating unit 302 at stepS201 acquires the storage picture of the directly previous field fromthe storage picture memory 303.

At the next step S202, the storage picture generating unit 302 detectsthe feature area, by the detection circuit 154, from the storage pictureof the N-fold density, supplied from the storage memory 153.

At the next step S203, the 1:N density motion vector is detected of thepupil picture based on the feature area detected at step S192.

At the next step S194, the storage picture from the storage memory 153is phase-shifted by the phase-shifting circuit 156, based on the motionvector detected at step S193, to effect position matching with respectto the pupil picture.

At the next step S195, the storage picture and the pupil picture,position-matched at step S194, are summed and synthesized together toform a new storage picture.

At the next step S196, the new storage picture, synthesized at the abovestep S195, is output.

At the next step S197, it is verified whether or not the totality of thepicture data for one field has been processed. Should there be anypicture data left for processing, the program reverts to step S191 torepeat the processing as from step S191 to S197 to process the totalityof the picture data to terminate the processing.

The processing for detecting the motion vector in case of generating apicture (storage picture) higher in picture quality than the inputpicture (pupil picture) is explained.

An instance of such processing is shown in FIG. 44A, illustratingdetection of a motion vector between a sole picture (base picture) Psand a picture Ph having four-fold density in the perpendicular directionwith respect to this picture Ps. In FIG. 44A, broken lines indicate theline positions, which in reality are lines devoid of pixels. The pictureof four-fold density Ph may be handled as four uni-fold density picturesPh1, Ph2, Ph3 and Ph4, of which the picture Ph1 is a picture spatiallycoincident with the base picture Ps and having line positions coincidentwith those of the base picture Ps, with the remaining three uni-folddensity pictures having the respective line positions sequentiallyshifted each by one line.

That is, in FIG. 44A, the picture Ph1 is obtained on selecting theuppermost one of four consecutive lines of a set on the high densitypicture Ph. The picture Ph2 is obtained on selecting the seconduppermost one of the four consecutive lines of the set on the highdensity picture Ph, while the picture Ph3 is obtained on selecting thesecond lowermost one of the four consecutive lines of the set. Thepicture Ph4 is obtained on selecting the lowermost one of the fourconsecutive lines of the set. A four-fold density picture is formed bycombining these four pictures Ph1 to Ph4 together.

First, a base block Bs and a reference block Bh1 of the same size and ofthe same shape as the base block, each being indicated as a 5×5 block inFIG. 44B, are set at spatially registering positions with respect to thepicture Ps and one of the four pictures Ph to Ph4, for example, Ph1,respectively, as shown in FIG. 44B. Then, absolute values of thedifferences of the values of pixels lying at the same positions as thoseof the base block Bs are found, as shown in FIG. 45. These absolutevalues are summed over the entire block to find the sum of the absolutevalues. The reference block Bh1 then is moved to different positions, interms of a pixel of the uni-fold density picture as a unit, to find thesum of the absolute values at each of the as-moved positions. The sumsof the absolute values, thus found, are stored in a table of the sums ofthe absolute values.

The range of movement is defined as a search range. For example, thesums of the absolute values are calculated for five reference blocks,shifted one pixel in the horizontal direction, and three referenceblocks, shifted one pixel in the vertical direction, totaling at 5×3reference blocks. In this case, a table of 5×3 sums of the absolutevalues is obtained. The center position of the 5×3 range is the point oforigin 0. The point of origin 0 coincides with the spatial center of thebase block Bs and that of the reference block Bh1. If, in the ultimatetable of the sums of the absolute values T0, as now explained, theposition of the reference block which gives the smallest value is at thepoint of origin, the motion vector is 0.

Next, a reference block Bh2, similar in shape and size to the referenceblock Bh1, is set at the same spatially position of the decimatedpicture Ph2 as the reference block Bh1. As in the case of the referenceblock Bh1, the sums of the absolute values of the differences betweenthe base block Bs and the reference block Bh2 are found to produce atable of the sums of the absolute values T2. This table T2 is spatiallybelow the table T1 by one line of the four-fold density picture. For thereference blocks Bh3, Bh4, block matching is effected with the baseblock Bs, as in the reference blocks Bh1, Bh2, to obtain tables of thesums of the absolute values T3 and T4. The table T3 is spatially belowthe table T2 by one line of the four-fold density picture, while thetable T4 is spatially below the table T3 by one line of the four-folddensity picture.

The four tables are synthesized in a reverse order to that in producingfour uni-fold density pictures from the four-fold density picture toproduce the ultimate table of the sums of the absolute values T0. Thistable shows the distribution of the sums of the absolute values of the5×3×4 blocks. The smallest sum value is detected in the table T0. Thevector from the point of origin 0 to the position of the reference blockwhich gives the smallest sum value is detected as being the motionvector. This achieves detection of the motion vector with a precision ofthe four-fold density picture.

Although the above-described embodiment is configured for detecting themotion vector between the base picture and a picture having thefour-fold density in the perpendicular direction, it is also possible todetect the motion vector between an N-fold density picture and the basepicture, where N is preferably an integer exceeding 2, in the horizontaldirection or in both the horizontal and vertical directions, in place ofin the vertical direction.

FIG. 47 shows the configuration of a motion vector detection device 400for detecting the motion vector between the base picture and thefour-fold density picture in the perpendicular direction. This motionvector detection device 400 includes a block forming circuit 401 forsplitting the base picture (picture Ps in FIG. 44A) into blocks and aphase separation circuit 402 for separating the four-fold densitypicture into N, for example, four, pictures (pictures Ph1 to Ph4 in FIG.44A) having the same density as that of the base picture and also havingthe same spatial relationship as the base picture. The phase separationcircuit 402 is fed with the base picture Ps and outputs data of thefirst line, through data of the second and third lines to data of thefourth line of each of sets of four lines, these data being thusseparated on the line basis from one another.

The output of the picture Ph1 of the phase separation circuit 402 issent to a block forming circuit 403 where it is broken down into blocks.Similarly, the outputs of the pictures Ph2, Ph3 and Ph4 of the phaseseparation circuit 402 are sent to block forming circuits 404 to 406,respectively, where the outputs of the respective pictures are brokendown into blocks. The block forming circuits 403 to 406 separate thepictures Ph1 to Ph4 each into blocks of the same size and shape as theblock formed by the block forming circuit 401, such as block of 5×5pixels.

An output of the block forming circuit 401 is sent to error detectioncircuits 407 to 410. An output of the error detection circuit 407 hasits other input fed with an output of the block forming circuit 403. Theerror detection circuit 407 summarized the sums of the absolute valuesof the differences of the values of pixels at the registering positionsof the block Bs of the base picture and the block of the picture Ph1from the block forming circuit 403. A table T1 for storing the sums ofthe absolute values is formed from one position of the reference blockto another.

The other input of the error detection circuit 408 is fed with an outputof the block forming circuit 404. The error detection circuit 408summarizes the sums of the absolute values of the differences of thepixel values of the pixels at the registering positions of the block ofthe base picture and the block of the picture Ph2 from the block formingcircuit 404. The error detection circuit 408 forms a table for holdingthe sums of the absolute values T2 from one position of the referenceblock to another. Similarly, the error detection circuit 409 forms atable T3 of the sums of the absolute values of the differences betweenthe block of the base picture and the block of the picture Ph3, whilethe error detection circuit 410 forms a table T4 of the sums of theabsolute values of the differences between the block of the base pictureand the block of the picture Ph4.

The tables T1 to T4, formed by the error detection circuits 407 to 410,are sent to a phase synthesis circuit 411. The phase synthesis circuit411 summarizes the sums of the absolute values in a reverse sequence tothat used in the phase separation in the phase separation circuit 402 toform the absolute table T0. A motion vector detection circuit 412detects the motion vector as it references thee table T0 formed by phasesynthesis circuit 411. That is, the motion vector detection circuit 412detects the least sum value among the sums of the absolute values tooutput a motion vector. In a lower part of the block diagram of FIG. 47,there is schematically shown the spatial relationship between pixels andsums of absolute valuers of the differences of the two pictures in theprocessing by the phase separation circuit 402, error detection circuits407 to 410 and in the motion vector detection circuit 412.

In the learning processing by the present picture processing apparatus300, as described above, not only the teacher picture and the pupilpicture but also the storage picture are referenced to calculate theprediction coefficients. In calculating the prediction coefficients, themovement of the subject pixel is also used as one of the feature valuesto effect classification of the subject pixel. As a result, if a subjectpixel is verified to be still, that is if a given picture is classifiedinto a class of a still picture, the picture information of the storagepicture, in which the noise of the still picture portion is reduced moreeffectively, is learned to a larger extent, whereas, if a given pictureis classified into a class of a moving picture, the picture informationof the storage picture, in which the noise of the moving picture portionis reduced more effectively, is learned to a larger extent.

With the above-described picture processing apparatus 100 or 190,executing the processing of picture generation using the predictioncoefficients calculated in the learning processing by this pictureprocessing apparatus 300, it becomes possible to reduce the noisecontained in the input picture or to correct the edge of the featurearea.

In the above-described embodiment, the prediction coefficientscalculated by the learning processing by the picture processingapparatus 300 are stored in the ROM tables 108, 196 of the pictureprocessing apparatus 100, 190 taking charge of the processing forpicture generation. It is, however, also possible to generate dataindicating pixel values, other coefficients or preset equations, neededto generate an output picture, and to have the so generated data storedin the ROM tables 108, 196.

In the above-described embodiment, a picture reduced in noise or apicture the edge of the feature area of which is not blurred isproduced. The present invention may, however, be also applied to thecase of generating a picture higher in picture quality than the inputpicture as to gradation, pixels or number of bits of data.

In the foregoing, the input picture is a progressive picture. Thepresent invention may, however, be also applied to such a case in whichthe input picture is an interlaced picture.

The above-described sequence of operations by the picture processingapparatus 100, 190, 300 may be carried out on the hardware, however, itmay also be carried out by the software. In the latter case, theabove-described picture processing apparatus 100 may be functionallyimplemented by having the program forming the software installed andexecuted on a computer.

FIG. 48 is a block diagram showing the configuration of an embodiment ofa computer 501 operating as the above-described picture processingapparatus 100, 190 or 300. To a CPU (central processing unit) 511 isconnected an input/output interface 516 over a bus 515. If a user'scommand is input from an input unit 518, such as a keyboard or a mouse,over an input/output interface 516, the program stored in a ROM(read-only memory) 512, or a recording medium, such as a hard disc 514,a magnetic disc 531, an optical disc 532, a magneto-optical disc 533, ora semiconductor memory 534, loaded on a drive 520, is loaded and run ona RAM (random access memory) 513. This permits the above-describedvarious processing operations to be carried out. The CPU 511 alsooutputs the results of the processing on a display unit 517, formede.g., by an LCD (liquid crystal display) through an input/outputinterface 516 as necessary. The program may also be stored on the harddisc 514 or on the ROM 512 of the computer 501 at the outset andfurnished in this state to the user. Alternatively, the program may befurnished as a packaged medium, such as magnetic disc 531, optical disc532, magneto-optical disc 533 or semiconductor memory 534. Stillalternatively, the program may also be furnished to the hard disc 514over a satellite or network through a communication unit 519.

In the present specification, the steps stating the program furnished bythe recording medium include not only the processing carried outchronologically in the sequence stated therein, but also such processingwhich is carried out in parallel or batch-wise, without necessarilybeing carried out chronologically.

In the above-described picture processing method and apparatus, and theprogram for a recording medium, according to the present invention, afirst picture is acquired, the so-acquired first picture is stored, anda new acquired first picture is stored at a position registering withthe so stored first picture. Based on the position of a subject pixel ofa second picture, the first pixel information is extracted from both thestorage picture and the first picture acquired, and a preset featurevalue is detected from the first pixel information. Based on thisfeature value, the subject pixel is classified to one of plural classesby way of classification. Based on the position of the subject pixel,the second pixel information is extracted from both the storage pictureand the acquired first picture and, using this second pixel information,the subject pixel is generated in accordance with a forming systempreset in association with the class obtained in the classification. So,the second picture higher in picture quality than the first picture maybe generated satisfactorily.

Moreover, in the picture processing method and apparatus, and theprogram for a recording medium, according to the present invention, anew pupil picture generated is stored at a position registering with astored pupil picture to store a storage picture of the pupil picture.Based on the position of a subject pixel of a teacher picture equivalentto the second picture, the first pixel information is extracted fromboth the storage picture and the generated pupil picture. From the firstpixel information, the preset feature value is detected and, based onthe so detected feature value, the subject pixel is classified to one ofthe plural classes, by way of classification. Based on the position ofthe subject pixel, the second pixel information is extracted from boththe storage picture and the generated first picture and preset data isfound from class to class by using the second pixel information and theteacher picture. In this manner, the second picture may be generatedwhich is higher in picture quality than the first picture.

1. A picture processing apparatus for generating a second picture from afirst picture, said second picture being of higher picture quality thansaid first picture, comprising: acquisition means for acquiring saidfirst picture; storage means for storing said first picture, acquired bysaid acquisition means; storage processing means for storing a new firstpicture acquired by said acquisition means at a position registeringwith said first picture stored in said storage means to permit a storagepicture of said first picture to be stored in said storage means; firstextraction means for extracting the first pixel information from bothsaid storage picture and said first picture, acquired by saidacquisition means, based on the position of a subject pixel of saidsecond picture; feature value detection means for detecting a presetfeature value from said first pixel information; classification meansfor classifying said subject pixel to one of a plurality of classesbased on said feature value; second extraction means for extracting thesecond pixel information from both said storage picture and said firstpicture, acquired by said acquisition means, based on the position ofsaid subject pixel; and generating means for generating said subjectpixel by using said second pixel information in accordance with agenerating system preset corresponding to the classes classified by saidclassification means.
 2. The picture processing apparatus according toclaim 1 wherein said storage processing means weight-sums said firstpicture and said storage picture in accordance with a preset weightingvalue to cause a resulting picture to be stored in said storage means.3. The picture processing apparatus according to claim 2 wherein saidstorage processing means detects the motion of said first picture andsets said weighting value based on the detected motion.
 4. The pictureprocessing apparatus according to claim 2 wherein said storage pictureis a picture higher in SN ratio in a still picture portion than saidfirst picture.
 5. The picture processing apparatus according to claim 1wherein said storage processing means detects the motion of a featurearea included in said first picture and causes said first picture to bemoved to and stored at a position corresponding to said detected motionto cause the resulting storage picture to be stored in said storagemeans.
 6. The picture processing apparatus according to claim 5 whereinthe pixel density of said storage picture is higher than that of saidfirst picture.
 7. The picture processing apparatus according to claim 5wherein the pixel density of said storage picture is equal to that ofsaid first picture.
 8. The picture processing apparatus according toclaim 5 wherein the number of said pixels of said storage picture islarger than that of said first picture.
 9. The picture processingapparatus according to claim 1 wherein said first extraction meansextracts class taps from said first picture as said first pixelinformation and wherein said feature value detection means detects thedistribution of pixel values of pixels forming said class tap as saidfeature value.
 10. The picture processing apparatus according to claim 1wherein said first extraction means extracts one or a plurality ofpixels from each of said first picture and said storage picture fromeach of said first picture and said storage picture; said feature valuedetection means making still/moving decision of said subject pixel byusing the pixel values of said one or a plurality of pixels; saidfeature value detection means detecting the results of decision as saidfeature value.
 11. The picture processing apparatus according to claim10 wherein said feature value detection means calculates the differencein luminance values of said first picture and said storage picture byusing the pixel values of said one or a plurality of pixels; saidfeature value detection means detecting the calculated results as saidfeature value.
 12. The picture processing apparatus according to claim 1wherein said first extraction means extracts a first class tap from saidfirst picture as said first pixel information and also extracts a secondclass tap from said storage picture; said feature value detection meansmaking still/moving decision of said subject pixel by using said firstand second class taps; said feature value detection means detecting theresults of decision and the distribution of pixel values of pixelsmaking up said first class tap as said feature values.
 13. The pictureprocessing apparatus according to claim 1 wherein said second extractionmeans extracts prediction taps as said second pixel information; saidgenerating means including coefficient storage means having storedtherein coefficients preset from class to class; said generating meansexecuting calculations using said coefficients and said prediction tapsto generate said second picture.
 14. A picture processing method forgenerating a second picture from a first picture, said second picturebeing of higher picture quality than said first picture, comprising: anacquisition step of acquiring said first picture; a storage step ofstoring said first picture, acquired by said acquisition step; a storageprocessing step of storing a new first picture acquired by saidacquisition step at a position registering with said first picturestored at said storage step to permit a storage picture of said firstpicture to be stored at said storage step; a first extraction step ofextracting the first pixel information from both said storage pictureand said first picture, acquired by said acquisition step, based on theposition of a subject pixel of said second picture; a feature valuedetection step of detecting a preset feature value from said first pixelinformation; a classification step of classifying said subject pixel toone of a plurality of classes based on said feature value; a secondextraction step of extracting the second pixel information from bothsaid storage picture and said first picture, acquired by saidacquisition step, based on the position of said subject pixel; and agenerating step of generating said subject pixel by said second pixelinformation in accordance with a generating system preset correspondingto the classes classified by said classification step.
 15. The pictureprocessing method according to claim 14 wherein said storage processingstep weight-sums said first picture and said storage picture inaccordance with a preset weighting value to cause a resulting picture tobe stored at said storage step.
 16. The picture processing methodaccording to claim 15 wherein said storage processing step detects themotion of said first picture and sets said weighting value based on thedetected motion.
 17. The picture processing method according to claim 15wherein said storage picture is a picture higher in SN ratio in a stillpicture portion than said first picture.
 18. The picture processingmethod according to claim 14 wherein said storage processing stepdetects the motion of a feature area included in said first picture andcauses said first picture to be moved to and stored at a positioncorresponding to said detected motion to cause the resulting storagepicture to be stored at said storage step.
 19. The picture processingmethod according to claim 18 wherein the pixel density of said storagepicture is higher than that of said first picture.
 20. The pictureprocessing method according to claim 18 wherein the pixel density ofsaid storage picture is equal to that of said first picture.
 21. Thepicture processing method according to claim 18 wherein the number ofsaid pixels of said storage picture is larger than that of said firstpicture.
 22. The picture processing method according to claim 14 whereinsaid first extraction step extracts class taps from said first pictureas said first pixel information and wherein said feature value detectionstep detects the distribution of pixel values of pixels forming saidclass tap as said feature value.
 23. The picture processing methodaccording to claim 14 wherein said first extraction step extracts one ora plurality of pixels from each of said first picture and said storagepicture as said first pixel information; said feature value detectionstep making still/moving decision of said subject pixel by using thepixel values of said one or a plurality of pixels; said feature valuedetection step detecting the results of decision as said feature value.24. The picture processing method according to claim 23 wherein saidfeature value detection step calculates the difference in luminancevalues of said first picture and said storage picture, by using thepixel values of said one or a plurality of pixels; said feature valuedetection step detecting the calculated results as said feature value.25. The picture processing method according to claim 14 wherein saidfirst extraction step extracts a first class tap from said first pictureas said first pixel information and also extracts a second class tapfrom said storage picture; said feature value detection step makingstill/moving decision of said subject pixel by using said first andsecond class taps; said feature value detection step detecting theresults of decision and the distribution of pixel values of pixelsmaking up said first class tap as said feature values.
 26. The pictureprocessing method according to claim 14 wherein said second extractionstep extracts prediction taps as said second pixel information; saidgenerating step executing calculations using said coefficients presetfrom class to class and said prediction taps to generate said secondpicture.
 27. A recording medium having recorded thereon acomputer-readable program adapted for generating a second picture from afirst picture, said second picture being of higher picture quality thansaid first picture, said program comprising: an acquisition step ofacquiring said first picture; a storage step of storing said firstpicture, acquired by said acquisition step; a storage processing step ofstoring a new first picture acquired by said acquisition step at aposition registering with said first picture stored at said storage stepto permit a storage picture of said first picture to be stored at saidstorage step; a first extraction step of extracting the first pixelinformation from both said storage picture and said first picture,acquired by said acquisition step, based on the position of a subjectpixel of said second picture; a feature value detection step ofdetecting a preset feature value from said first pixel information; aclassification step of classifying said subject pixel to one of aplurality of classes based on said feature value; a second extractionstep of extracting the second pixel information from both said storagepicture and said first picture, acquired by said acquisition step, basedon the position of said subject pixel; and a generating step ofgenerating said subject pixel by said second pixel information inaccordance with a generating system preset corresponding to the classesclassified at the classification step.
 28. The recording medium having acomputer-readable program recorded thereon according to claim 27 whereinsaid storage processing step weight-sums said first picture and saidstorage picture in accordance with a preset weighting value to cause aresulting picture to be stored at said storage step.
 29. The recordingmedium having a computer-readable program recorded thereon according toclaim 28 wherein said storage processing step detects the motion of saidfirst picture and sets said weighting value based on the detectedmotion.
 30. The recording medium having a computer-readable programrecorded thereon according to claim 28 wherein said storage picture is apicture higher in SN ratio in a still picture portion than said firstpicture.
 31. The recording medium having a computer-readable programrecorded thereon according to claim 27 wherein said storage processingstep detects the motion of a feature area included in said first pictureand causes said first picture to be moved to and stored at a positioncorresponding to said detected motion to cause the resulting storagepicture to be stored at said storage step.
 32. The recording mediumhaving a computer-readable program recorded thereon according to claim28 wherein the pixel density of said storage picture is higher than thatof said first picture.
 33. The recording medium having acomputer-readable program recorded thereon according to claim 28 whereinthe pixel density of said storage picture is equal to that of said firstpicture.
 34. The recording medium having a computer-readable programrecorded thereon according to claim 28 wherein the number of pixels ofsaid storage picture is larger than that of said first picture.
 35. Therecording medium having a computer-readable program recorded thereonaccording to claim 27 wherein said first extraction step extracts classtaps from said first picture as said first pixel information and whereinsaid feature value detection step detects the distribution of pixelvalues of pixels forming said class tap as said feature value.
 36. Therecording medium having a computer-readable program recorded thereonaccording to claim 27 wherein said first extraction step extracts one ora plurality of pixels from each of said first picture and said storagepicture as said first pixel information; said feature value detectionstep making still/moving decision of said subject pixel by using thepixel values of said one or a plurality of pixels; said feature valuedetection step detecting the results of decision as said feature value.37. The recording medium having a computer-readable program recordedthereon according to claim 36 wherein said feature value detection stepcalculates the difference in luminance values of said first picture andsaid storage picture by using the pixel values of said one or aplurality of pixels; said feature value detection step detecting thecalculated results as said feature value.
 38. The recording mediumhaving a computer-readable program recorded thereon according to claim27 wherein said first extraction step extracts a first class tap fromsaid first picture as said first pixel information and also extracts asecond class tap from said storage picture; said feature value detectionstep making still/moving decision of said subject pixel by using saidfirst and second class taps; said feature value detection step detectingthe results of decision and the distribution of pixel values of pixelsmaking up said first class tap as said feature values.
 39. The recordingmedium having a computer-readable program recorded thereon according toclaim 27 wherein said second extraction step extracts prediction taps assaid second pixel information; said generating step executingcalculations using said coefficients preset from class to class and saidprediction taps to generate said second picture.
 40. A pictureprocessing apparatus for learning preset data used in generating asecond picture from a first picture, said second picture being higher inpicture quality than said first picture; comprising: generating meansfor generating a pupil picture equivalent to said first picture; storagemeans for storing said pupil picture; storage processing means forcausing a new pupil picture, generated by said generating means, to bestored at a position registering with said pupil picture stored in saidstorage means for causing a storage picture of said pupil picture to bestored in said storage means; first extraction means for extracting saidfirst picture information from both said storage picture and said pupilpicture generated by said generating means, based on the position ofsaid subject pixel of teacher data equivalent to said second picture;feature value detection means for detecting a preset feature value fromsaid first pixel information; classification means for classifying saidsubject pixel to one of a plurality of classes; second extraction meansfor extracting the second pixel information from both said storagepicture and the first picture generated by said generating means, basedon the position of said subject pixel; and calculation means for findingsaid preset data from one class of classifying by said classificationmeans to another, by using said second pixel information and saidteacher data.
 41. The picture processing apparatus according to claim 40wherein said storage processing means weight-adds said pupil picture andsaid storage picture in accordance with a preset weighting value andcauses the resulting picture to be stored in said storage means.
 42. Thepicture processing apparatus according to claim 41 wherein said storagemeans sets said weighting value based on the values of said pupilpicture and the storage picture or the difference values thereof. 43.The picture processing apparatus according to claim 41 wherein saidstorage processing means detects the motion of said pupil picture andsets said weighting value based on the detected motion.
 44. The pictureprocessing apparatus according to claim 41 wherein said storage pictureis a picture higher in the SN ratio in a still picture portion thereof.45. The picture processing apparatus according to claim 42 wherein saidstorage processing means detects the motion of a feature area containedin said pupil area and causes said pupil picture to be moved to andstored at a position corresponding to the detected motion to cause theresulting storage picture to be stored in said storage means.
 46. Thepicture processing apparatus according to claim 45 wherein the pixeldensity of said storage picture is higher than that of said pupilpicture.
 47. The picture processing apparatus according to claim 45wherein the pixel density of said storage picture is equal to that ofsaid pupil picture.
 48. The picture processing apparatus according toclaim 40 wherein said first extraction means extracts a class tap fromsaid pupil picture as said first pixel information and wherein saidfeature value detection means detects the distribution of pixel valuesof pixels making up said class tap as said feature value.
 49. Thepicture processing apparatus according to claim 40 wherein said firstextraction means extracts one or a plurality of pixels from each of saidpupil picture and the storage picture as said first pixel informationand wherein said feature value detection means uses the pixel values ofsaid one or plural pixels to make still/moving decision of said subjectpixel, while detecting the results of decision as said feature value.50. The picture processing apparatus according to claim 49 wherein saidfirst feature value detection means detects the differences of luminancevalues of said first picture and said storage picture by using the pixelvalue of said one or plural pixels and detects the calculated results assaid feature value.
 51. The picture processing apparatus according toclaim 40 wherein said first extraction means extracts a first class tapfrom said pupil picture as said first pixel information, whileextracting a second class tap from said storage picture; and whereinsaid feature value detection means makes still/moving decision of saidsubject pixel by using said first and second class taps and detects theresults of decision and the distribution of pixel values of pixelsmaking up said first class tap as said feature value.
 52. The pictureprocessing apparatus according to claim 40 wherein said secondextraction means extracts prediction taps as said second pixelinformation.
 53. A picture processing method by a picture processingapparatus for learning preset data used in generating a second picturefrom a first picture, said second picture being higher in picturequality than said first picture; comprising: a generating step ofgenerating a pupil picture equivalent to said first picture; a storagestep of storing said pupil picture; a storage processing step of causinga new pupil picture, generated by processing at said generating step, tobe stored at a position registering with said pupil picture stored insaid storage step for causing a storage picture of said pupil picture tobe stored in said storage step; a first extraction step of extractingsaid first picture information from both said storage picture and saidpupil picture generated by said generating step, based on the positionof said subject pixel of teacher data equivalent to said second picture;a feature value detection step of detecting a preset feature value fromsaid first pixel information; a classification step of classifying saidsubject pixel to one of a plurality of classes, by way ofclassification, based on said feature value; a second extraction step ofextracting the second pixel information from both said storage pictureand the first picture generated by processing at said generating step,based on the position of said subject pixel; and a calculation step offinding said preset data from one class of classifying by saidclassification step to another, by using said second pixel informationand said teacher data.
 54. The picture processing method according toclaim 53 wherein said storage processing means weight-adds said pupilpicture and said storage picture in accordance with a preset weightingvalue and causes the resulting picture to be stored in said storagestep.
 55. The picture processing method according to claim 54 whereinsaid storage means sets said weighting value based on the values of saidpupil picture and the storage picture or the difference values thereof.56. The picture processing method according to claim 54 wherein saidstorage processing means detects the motion of said pupil picture andsets said weighting value based on the detected motion.
 57. The pictureprocessing method according to claim 54 wherein said storage picture isa picture higher in the SN ratio in a still picture portion thereof thansaid pupil picture.
 58. The picture processing method according to claim55 wherein said storage processing means detects the motion of a featurearea contained in said pupil area and causes said pupil picture to bemoved to and stored at a position corresponding to the detected motionto cause the resulting storage picture to be stored in said storagestep.
 59. The picture processing method according to claim 58 whereinthe pixel density of said storage picture is higher than that of saidpupil picture.
 60. The picture processing method according to claim 58wherein the pixel density of said storage picture is equal to that ofsaid pupil picture.
 61. The picture processing method according to claim53 wherein said first extraction means extracts a class tap from saidpupil picture as said first pixel information and wherein said featurevalue detection means detects the distribution of pixel values of pixelsmaking up said class tap as said feature value.
 62. The pictureprocessing method according to claim 53 wherein said first extractionmeans extracts one or a plurality of pixels from each of said pupilpicture and the storage picture as said first pixel information andwherein said feature value detection means uses the pixel values of saidone or plural pixels to make still/moving decision of said subjectpixel, while detecting the results of decision as said feature value.63. The picture processing method according to claim 62 wherein saidfirst feature value detection means detects the differences of luminancevalues of said first picture and said storage picture by using the pixelvalue of said one or plural pixels and detects the calculated results assaid feature value.
 64. The picture processing method according to claim53 wherein said first extraction means extracts a first class tap fromsaid pupil picture as said first pixel information, while extracting asecond class tap from said storage picture; and wherein said featurevalue detection means makes still/moving decision of said subject pixelby using said first and second class taps and detects the results ofdecision and the distribution of pixel values of pixels making up saidfirst class tap as said feature value.
 65. The picture processing methodaccording to claim 53 wherein said second extraction means extractsprediction taps as said second pixel information.
 66. A recording mediumhaving recorded thereon a computer-readable program for a pictureprocessing apparatus adapted for learning preset data usable forgenerating a second picture from a first picture, said second picturebeing of higher picture quality than said first picture, said programcomprising: a generating step of generating a pupil picture equivalentto said first picture; a storage step of storing said pupil picture; astorage processing step of causing a new pupil picture, generated byprocessing at said generating step, to be stored at a positionregistering with said pupil picture stored in said storage step ofcausing a storage picture of said pupil picture to be stored in saidstorage step; a first extraction step of extracting said first pictureinformation from both said storage picture and said pupil picturegenerated by said generating step, based on the position of a subjectpixel of teacher data equivalent to said second picture; a feature valuedetection step of detecting a preset feature value from said first pixelinformation; a classification step of classifying said subject pixel toone of a plurality of classes; a second extraction step of extractingthe second pixel information from both said storage picture and thefirst picture generated by processing at said generating step, based onthe position of said subject pixel; and a calculation step of findingsaid preset data from one class of classifying by said classificationstep to another, by using said second pixel information and said teacherdata.
 67. The recording medium having recorded thereon acomputer-readable program according to claim 66 wherein said storageprocessing step weight-adds said pupil picture and said storage picturein accordance with a preset weighting value and causes the resultingpicture to be stored at said storage step.
 68. The recording mediumhaving recorded thereon a computer-readable program according to claim67 wherein said storage step sets said weighting value based on thevalues of said pupil picture and the storage picture or the differencevalues thereof.
 69. The recording medium having recorded thereon acomputer-readable program according to claim 67 wherein said storageprocessing step detects the motion of said pupil picture and sets saidweighting value based on the detected motion.
 70. The recording mediumhaving recorded thereon a computer-readable program according to claim67 wherein said storage picture is a picture higher in the SN ratio in astill picture portion thereof than said pupil picture.
 71. The recordingmedium having recorded thereon a computer-readable program according toclaim 68 wherein said storage processing step detects the motion of afeature area contained in said pupil area and causes said pupil pictureto be moved to and stored at a position corresponding to the detectedmotion to cause the resulting storage picture to be stored at saidstorage step.
 72. The recording medium having recorded thereon acomputer-readable program according to claim 71 wherein the pixeldensity of said storage picture is higher than that of said pupilpicture.
 73. The recording medium having recorded thereon acomputer-readable program according to claim 71 wherein the pixeldensity of said storage picture is equal to that of said pupil picture.74. The recording medium having recorded thereon a computer-readableprogram according to claim 66 wherein said first extraction stepextracts a class tap from said pupil picture as said first pixelinformation and wherein said feature value detection step detects thedistribution of pixel values of pixels making up said class tap as saidfeature value.
 75. The recording medium having recorded thereon acomputer-readable program according to claim 66 wherein said firstextraction step extracts one or a plurality of pixels from each of saidpupil picture and the storage picture, as said first pixel information,and wherein said feature value detection step uses the pixel values ofsaid one or plural pixels to make still/moving decision of said subjectpixel, while detecting the results of decision as said feature value.76. The recording medium having recorded thereon a computer-readableprogram according to claim 75 wherein said first feature value detectionstep detects the differences of luminance values of said first pictureand said storage picture by using the pixel value of said one or pluralpixels and detects the calculated results as said feature value.
 77. Therecording medium having recorded thereon a computer-readable programaccording to claim 66 wherein said first extraction step extracts afirst class tap from said pupil picture as said first pixel information,while extracting a second class tap from said storage picture; andwherein said feature value detection step makes still/moving decision ofsaid subject pixel by using said first and second class taps and detectsthe results of decision and the distribution of pixel values of pixelsmaking up said first class tap as said feature value.
 78. The recordingmedium having recorded thereon a computer-readable program according toclaim 66 wherein said second extraction step extracts prediction taps assaid second pixel information.